Configuration · ArchiveBox/ArchiveBox Wiki · GitHub (repo_wiki) (https://github.githubassets.com/) (https://github.com/opensearch.xml) (GitHub) (https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#public_index--public_snapshots--public_add_view)  Skip to content     Navigation Menu    (https://github.com/)    (https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2FArchiveBox%2FArchiveBox%2Fwiki%2FConfiguration) Sign in    Product    (https://github.com/features/actions)   Actions Automate any workflow    (https://github.com/features/packages)   Packages Host and manage packages    (https://github.com/features/security)    Security Find and fix vulnerabilities    (https://github.com/features/codespaces)    Codespaces Instant dev environments    (https://github.com/features/copilot)    GitHub Copilot Write better code with AI    (https://github.com/features/code-review)    Code review Manage code changes    (https://github.com/features/issues)   Issues Plan and track work    (https://github.com/features/discussions)    Discussions Collaborate outside of code      Explore (https://github.com/features) All features   (https://docs.github.com/) Documentation     (https://skills.github.com/) GitHub Skills     (https://github.blog/) Blog         Solutions    By size (https://github.com/enterprise) Enterprise   (https://github.com/team) Teams   (https://github.com/enterprise/startups) Startups     By industry (https://github.com/solutions/industries/healthcare) Healthcare   (https://github.com/solutions/industries/financial-services) Financial services   (https://github.com/solutions/industries/manufacturing) Manufacturing     By use case (https://github.com/solutions/ci-cd) CI/CD & Automation   (https://github.com/solutions/devops) DevOps   (https://github.com/solutions/devsecops) DevSecOps       Resources    Topics (https://github.com/resources/articles/ai) AI   (https://github.com/resources/articles/devops) DevOps   (https://github.com/resources/articles/security) Security   (https://github.com/resources/articles/software-development) Software Development   (https://github.com/resources/articles) View all     Explore (https://resources.github.com/learn/pathways) Learning Pathways     (https://resources.github.com/) White papers, Ebooks, Webinars     (https://github.com/customer-stories) Customer Stories   (https://partner.github.com/) Partners         Open Source    (https://github.com/sponsors) GitHub Sponsors Fund open source developers      (https://github.com/readme) The ReadME Project GitHub community articles      Repositories (https://github.com/topics) Topics   (https://github.com/trending) Trending   (https://github.com/collections) Collections       Enterprise    (https://github.com/enterprise)     Enterprise platform AI-powered developer platform      Available add-ons (https://github.com/enterprise/advanced-security)    Advanced Security Enterprise-grade security features    (https://github.com/features/copilot#enterprise)    GitHub Copilot Enterprise-grade AI features    (https://github.com/premium-support)    Premium Support Enterprise-grade 24/7 support        (https://github.com/pricing) Pricing    (Search or jump to...)    Search or jump to...                       (https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2FArchiveBox%2FArchiveBox%2Fwiki%2FConfiguration) Sign in   (https://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fwiki%2Fshow&source=header-repo&source_repo=ArchiveBox%2FArchiveBox) Sign up  Reseting focus               (https://github.com/ArchiveBox) ArchiveBox   / (https://github.com/ArchiveBox/ArchiveBox) ArchiveBox   Public     Sponsor        (https://github.com/login?return_to=%2FArchiveBox%2FArchiveBox)   Notifications  You must be signed in to change notification settings  (https://github.com/login?return_to=%2FArchiveBox%2FArchiveBox)   Fork (1,113) 1.1k   (https://github.com/login?return_to=%2FArchiveBox%2FArchiveBox)   Star  (20,696) 20.7k        (https://github.com/ArchiveBox/ArchiveBox)   Code (Not available)    (https://github.com/ArchiveBox/ArchiveBox/issues)    Issues (162) 162   (https://github.com/ArchiveBox/ArchiveBox/pulls)   Pull requests (3) 3   (https://github.com/ArchiveBox/ArchiveBox/discussions)   Discussions (Not available)    (https://github.com/ArchiveBox/ArchiveBox/actions)   Actions (Not available)    (https://github.com/ArchiveBox/ArchiveBox/projects)   Projects (1) 1   (https://github.com/ArchiveBox/ArchiveBox/wiki)   Wiki (Not available)    (https://github.com/ArchiveBox/ArchiveBox/security)   Security (1) 1   (https://github.com/ArchiveBox/ArchiveBox/pulse)   Insights (Not available)        Additional navigation options       Configuration Jump to bottom   Nick Sweeting edited this page (May 9, 2024, 1:02 AM UTC) May 9, 2024 · (https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration/_history) 160 revisions   Configuration     Configuration of ArchiveBox is done by using the archivebox config command, modifying the ArchiveBox.conf file in the data folder, or by using environment variables. All three methods work equivalently when using Docker as well. Some equivalent examples of setting some configuration options:  archivebox config --set CHROME_BINARY=google-chrome-stable # OR echo " CHROME_BINARY=google-chrome-stable"  >> ArchiveBox.conf # OR env CHROME_BINARY=google-chrome-stable archivebox add ~ /Downloads/bookmarks_export.html (archivebox config --set CHROME_BINARY=google-chrome-stable
# OR
echo "CHROME_BINARY=google-chrome-stable" >> ArchiveBox.conf
# OR
env CHROME_BINARY=google-chrome-stable archivebox add ~/Downloads/bookmarks_export.html)         Environment variables take precedence over the config file, which is useful if you only want to use a certain option temporarily during a single run. For more examples see (https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#run-archivebox-with-configuration-options) Usage: Configuration ... Available Configuration Options:  General Settings: Archiving process, output format, and timing. Archive Method Toggles: On/off switches for methods. Archive Method Options: Method tunables and parameters. Shell Options: Format & behavior of CLI output. Dependency Options: Specify exact paths to dependencies.  In case this document is ever out of date, it's recommended to read the code that loads the config directly: (https://github.com/ArchiveBox/ArchiveBox/blob/master/archivebox/config.py#L27) archivebox/config.py  ➡️  General Settings     General options around the archiving process, output format, and timing.  OUTPUT_PERMISSIONS      Possible Values: [755 ]/644 /...Permissions to set the output directory and file contents to. This is useful when running ArchiveBox inside Docker as root and you need to explicitly set the permissions to something that the users on the host can access. Related options: PUID / PGID      PUID / PGID      Possible Values: [911 ]/1000 /... User and Group ownership to set the output directory and file contents to. Only settable as environment variables when using ArchiveBox in Docker. This is useful on some Docker setups when you want the data dir to be owned by the same UID/GID on the host and inside the container. PUID=0 is not allowed ((https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#do-not-run-as-root) do not run as root ), and PGID=0 is allowed but not recommended. PUID s and PGID s below 100 cause many issues because they're often (https://github.com/ArchiveBox/ArchiveBox/discussions/1366) already in use by an existing linux user in docker, if the files must be owned by a low value ID e.g. 33 (www-data ), you may need to use (https://github.com/clecherbauer/docker-volume-bindfs) bindfs  to remap the permissions. Make sure if using NFS/SMB/FUSE that the volume allows setting ownership on files (e.g. don't set root_squash or all_squash on NFS shares). Learn more:  (https://docs.linuxserver.io/general/understanding-puid-and-pgid/) https://docs.linuxserver.io/general/understanding-puid-and-pgid/  (https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting#docker-permissions-issues) https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting#docker-permissions-issues  (https://github.com/ArchiveBox/ArchiveBox/issues/1304) https://github.com/ArchiveBox/ArchiveBox/issues/1304  (https://github.com/ArchiveBox/ArchiveBox/discussions/1366) https://github.com/ArchiveBox/ArchiveBox/discussions/1366  (https://github.com/ArchiveBox/ArchiveBox/blob/main/bin/docker_entrypoint.sh) https://github.com/ArchiveBox/ArchiveBox/blob/main/bin/docker_entrypoint.sh   Related options: OUTPUT_PERMISSIONS   ONLY_NEW      Possible Values: [True ]/False Toggle whether or not to attempt rechecking old links when adding new ones, or leave old incomplete links alone and only archive the new links. By default, ArchiveBox will only archive new links on each import. If you want it to go back through all links in the index and download any missing files on every run, set this to False . Note: Regardless of how this is set, ArchiveBox will never re-download sites that have already succeeded previously. When this is False it only attempts to fix previous pages have missing archive extractor outputs, it does not re-archive pages that have already been successfully archived.  TIMEOUT      Possible Values: [60 ]/120 /...Maximum allowed download time per archive method for each link in seconds.  If you have a slow network connection or are seeing frequent timeout errors, you can raise this value. Note: Do not set this to anything less than 15 seconds as it will cause Chrome to hang indefinitely and many sites to fail completely.  MEDIA_TIMEOUT      Possible Values: [3600 ]/120 /...Maximum allowed download time for fetching media when SAVE_MEDIA=True in seconds.  This timeout is separate and usually much longer than TIMEOUT because media downloaded with youtube-dl can often be quite large and take many minutes/hours to download.  Tweak this setting based on your network speed and maximum media file size you plan on downloading. Note: Do not set this to anything less than 10 seconds as it can often take 5-10 seconds for youtube-dl just to parse the page before it starts downloading media files.  Related options: SAVE_MEDIA     ADMIN_USERNAME / ADMIN_PASSWORD      Possible Values: [None ]/"admin" /... Only used on first run / initial setup in Docker. ArchiveBox will create an admin user with the specified username and password when these options are found in the environment.
Useful for setting up a Docker instance of ArchiveBox without needing to run a shell command to create the admin user. Equivalent to: $ archivebox manage createsuperuser
Username: < ADMIN_USERNAME> Password: < ADMIN_PASSWORD> Password (again): < ADMIN_PASSWORD>  ($ archivebox manage createsuperuser
Username: <ADMIN_USERNAME>
Password: <ADMIN_PASSWORD>
Password (again): <ADMIN_PASSWORD>)         More info: (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Authentication) https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Authentication  (https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#configuration) https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#configuration   Related options: PUBLIC_INDEX / PUBLIC_SNAPSHOTS / PUBLIC_ADD_VIEW      PUBLIC_INDEX / PUBLIC_SNAPSHOTS / PUBLIC_ADD_VIEW      Possible Values: [True ]/False Configure whether or not login is required to use each area of ArchiveBox. archivebox manage createsuperuser # set a password before disabling public access # these are the default values archivebox config - - set PUBLIC_INDEX = True # True = allow users to view main snapshots list without logging in archivebox config - - set PUBLIC_SNAPSHOTS = True # True = allow users to view snapshot content without logging in archivebox config - - set PUBLIC_ADD_VIEW = False # True = allow users to submit new URLs to archive without logging in  (archivebox manage createsuperuser  # set a password before disabling public access

# these are the default values
archivebox config --set PUBLIC_INDEX=True        # True = allow users to view main snapshots list without logging in
archivebox config --set PUBLIC_SNAPSHOTS=True    # True = allow users to view snapshot content without logging in
archivebox config --set PUBLIC_ADD_VIEW=False    # True = allow users to submit new URLs to archive without logging in)         More info: (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Authentication) https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Authentication  (https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#ui-usage) https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#ui-usage   CUSTOM_TEMPLATES_DIR      Possible Values: [None ]//path/to/custom_templates /... Path to a directory containing custom html/css/images for overriding the default UI styling.  Files found in the folder at the specified path can override any of the defaults in the (https://github.com/ArchiveBox/ArchiveBox/tree/dev/archivebox/templates) TEMPLATES_DIR  directory (copy files from that default dir into your custom dir to get started making a custom theme). If you've used django before, this works exactly the same way that django template overrides work (because it uses django under the hood). pip show -f archivebox | grep Location: | awk ' {print $2}'  # /opt/homebrew/lib/python3.11/site-packages pip show -f archivebox | grep archivebox/templates # archivebox/templates/admin/app_index.html # archivebox/templates/admin/base.html # archivebox/templates/admin/login.html # ... # copy default templates into a directory somewhere, edit as needed, then point archivebox to it, e.g. cp -r /opt/homebrew/lib/python3.11/site-packages/archivebox/templates ~ /archivebox/custom_templates
archivebox config --set CUSTOM_TEMPLATES_DIR=~ /archivebox/data/custom_templates (pip show -f archivebox | grep Location: | awk '{print $2}'
# /opt/homebrew/lib/python3.11/site-packages

pip show -f archivebox | grep archivebox/templates
# archivebox/templates/admin/app_index.html
# archivebox/templates/admin/base.html
# archivebox/templates/admin/login.html
# ...

# copy default templates into a directory somewhere, edit as needed, then point archivebox to it, e.g.
cp -r /opt/homebrew/lib/python3.11/site-packages/archivebox/templates ~/archivebox/custom_templates
archivebox config --set CUSTOM_TEMPLATES_DIR=~/archivebox/data/custom_templates)         Related options: FOOTER_INFO   REVERSE_PROXY_USER_HEADER      Possible Values: [Remote-User ]/X-Remote-User /... HTTP header containing user name from authenticated proxy. More info: (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Authentication) https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Authentication  (https://github.com/ArchiveBox/ArchiveBox/pull/866) https://github.com/ArchiveBox/ArchiveBox/pull/866   Related options: REVERSE_PROXY_WHITELIST  , LOGOUT_REDIRECT_URL   REVERSE_PROXY_WHITELIST      Possible Values: [<empty string> ],172.16.0.0/16 ,2001:d80::/26 /... Comma separated list of IP CIDRs which are allowed to use reverse proxy authentication. Both IPv4 and IPv6 IPs can be used next to each other. Empty string means "deny all". More info: (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Authentication) https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Authentication  (https://github.com/ArchiveBox/ArchiveBox/pull/866) https://github.com/ArchiveBox/ArchiveBox/pull/866   Related options: REVERSE_PROXY_USER_HEADER  , LOGOUT_REDIRECT_URL   LOGOUT_REDIRECT_URL      Possible Values: [/ ]/https://example.com/some/other/app /... URL to redirect users back to on logout when using reverse proxy authentication. More info: (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Authentication) https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Authentication  (https://github.com/ArchiveBox/ArchiveBox/pull/866) https://github.com/ArchiveBox/ArchiveBox/pull/866   Related options: REVERSE_PROXY_USER_HEADER  , REVERSE_PROXY_WHITELIST   LDAP      Possible Values: [False ]/True  Whether to use an external (https://jumpcloud.com/blog/what-is-ldap-authentication) LDAP server for authentication (e.g. OpenLDAP, MS Active Directory, OpenDJ, etc.). # first, install optional ldap addon to use this feature pip install archivebox[ldap] (# first, install optional ldap addon to use this feature
pip install archivebox[ldap])         Then set these configuration values to finish configuring LDAP: LDAP : True LDAP_SERVER_URI : " ldap://ldap.example.com:3389"  LDAP_BIND_DN : " ou=archivebox,ou=services,dc=ldap.example.com"  LDAP_BIND_PASSWORD : " secret-bind-user-password"  LDAP_USER_BASE : " ou=users,ou=archivebox,ou=services,dc=ldap.example.com"  LDAP_USER_FILTER : " (objectClass=user)"  LDAP_USERNAME_ATTR : " uid"  LDAP_FIRSTNAME_ATTR : " givenName"  LDAP_LASTNAME_ATTR : " sn"  LDAP_EMAIL_ATTR : " mail"   (LDAP: True
LDAP_SERVER_URI: "ldap://ldap.example.com:3389"
LDAP_BIND_DN: "ou=archivebox,ou=services,dc=ldap.example.com"
LDAP_BIND_PASSWORD: "secret-bind-user-password"
LDAP_USER_BASE: "ou=users,ou=archivebox,ou=services,dc=ldap.example.com"
LDAP_USER_FILTER: "(objectClass=user)"

LDAP_USERNAME_ATTR: "uid"
LDAP_FIRSTNAME_ATTR: "givenName"
LDAP_LASTNAME_ATTR: "sn"
LDAP_EMAIL_ATTR: "mail")         More info: (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Authentication) https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Authentication  (https://github.com/ArchiveBox/ArchiveBox/pull/1214) https://github.com/ArchiveBox/ArchiveBox/pull/1214  (https://github.com/django-auth-ldap/django-auth-ldap#example-configuration) https://github.com/django-auth-ldap/django-auth-ldap#example-configuration  (https://jumpcloud.com/blog/what-is-ldap-authentication) https://jumpcloud.com/blog/what-is-ldap-authentication   SNAPSHOTS_PER_PAGE      Possible Values: [40 ]/100 /... Maximum number of Snapshots to show per page on Snapshot list pages. Lower this value on slower machines to make the UI faster. Related options: SEARCH_BACKEND_TIMEOUT   FOOTER_INFO      Possible Values: [Content is hosted for personal archiving purposes only.  Contact server owner for any takedown requests. ]/Operated by ACME Co. /...Some text to display in the footer of the archive index.  Useful for providing server admin contact info to respond to takedown requests. Related options: TEMPLATES_DIR    URL_DENYLIST      Possible Values: [\.(css|js|otf|ttf|woff|woff2|gstatic\.com|googleapis\.com/css)(\?.*)?$ ]/.+\.exe$ /http(s)?:\/\/(.+)?example.com\/.* /... A regex expression used to exclude certain URLs from archiving.  You can use if there are certain domains, extensions, or other URL patterns that you want to ignore whenever they get imported.  Blacklisted URLs wont be included in the index, and their page content wont be archived. When building your exclusion list, you can check whether a given URL matches your regex expression in python like so: >> > import re >> > URL_DENYLIST = r'^http(s)?:\/\/(.+\.)?(youtube\.com)|(amazon\.com)\/.*$' # replace this with your regex to test >> > URL_DENYLIST_PTN = re .compile (URL_DENYLIST , re .IGNORECASE | re .UNICODE | re .MULTILINE ) >> > bool (URL_DENYLIST_PTN .search ('https://test.youtube.com/example.php?abc=123' )) # replace this with the URL to test True # this URL would not be archived because it matches the exclusion pattern  (>>> import re
>>> URL_DENYLIST = r'^http(s)?:\/\/(.+\.)?(youtube\.com)|(amazon\.com)\/.*$'  # replace this with your regex to test
>>> URL_DENYLIST_PTN = re.compile(URL_DENYLIST, re.IGNORECASE | re.UNICODE | re.MULTILINE)

>>> bool(URL_DENYLIST_PTN.search('https://test.youtube.com/example.php?abc=123'))  # replace this with the URL to test
True   # this URL would not be archived because it matches the exclusion pattern)         Note: all assets required to render each page are still archived, URL_DENYLIST /URL_ALLOWLIST do not apply to images, css, video, etc. visible inline within the page.  Note 2: These options used to be called URL_WHITELIST  & URL_BLACKLIST  before (https://github.com/ArchiveBox/ArchiveBox/releases) v0.7.1  . Related options: URL_ALLOWLIST  , SAVE_MEDIA  , SAVE_GIT  , GIT_DOMAINS    URL_ALLOWLIST      Possible Values: [None ]/^http(s)?:\/\/(.+)?example\.com\/?.*$ /... A regex expression used to exclude all URLs that don't match the given pattern from archiving.  You can use if there are certain domains, extensions, or other URL patterns that you want to restrict the scope of archiving to (e.g. to only archive a single domain, subdirectory, or filetype, etc.. When building your whitelist, you can check whether a given URL matches your regex expression in python like so: >> > import re >> > URL_ALLOWLIST = r'^http(s)?:\/\/(.+)?example\.com\/?.*$' # replace this with your regex to test >> > URL_ALLOWLIST_PTN = re .compile (URL_ALLOWLIST , re .IGNORECASE | re .UNICODE | re .MULTILINE ) >> > bool (URL_ALLOWLIST_PTN .search ('https://test.example.com/example.php?abc=123' )) True # this URL would be archived >> > bool (URL_ALLOWLIST_PTN .search ('https://test.youtube.com/example.php?abc=123' )) False # this URL would be excluded from archiving  (>>> import re
>>> URL_ALLOWLIST = r'^http(s)?:\/\/(.+)?example\.com\/?.*$'  # replace this with your regex to test
>>> URL_ALLOWLIST_PTN = re.compile(URL_ALLOWLIST, re.IGNORECASE | re.UNICODE | re.MULTILINE)

>>> bool(URL_ALLOWLIST_PTN.search('https://test.example.com/example.php?abc=123'))
True      # this URL would be archived

>>> bool(URL_ALLOWLIST_PTN.search('https://test.youtube.com/example.php?abc=123'))
False     # this URL would be excluded from archiving)         This option is useful for recursive archiving of all the pages under a given domain or subfolder (aka crawling/spidering), without following links to external domains / parent folders. # temporarily enforce a whitelist by setting the option as an environment variable export URL_ALLOWLIST=' ^http(s)?:\/\/(.+)?example\.com\/?.*$'  # then run your archivebox commands in the same shell archivebox add --depth=1 ' https://example.com'  archivebox list https://example.com | archivebox add --depth=1
archivebox list https://example.com | archivebox add --depth=1
archivebox list https://example.com | archivebox add --depth=1 # repeat up to desired depth ... # all URLs that don't match *.example.com will be excluded, e.g. a link to youtube.com would not be followed  (# temporarily enforce a whitelist by setting the option as an environment variable
export URL_ALLOWLIST='^http(s)?:\/\/(.+)?example\.com\/?.*$'

# then run your archivebox commands in the same shell
archivebox add --depth=1 'https://example.com'
archivebox list https://example.com | archivebox add --depth=1
archivebox list https://example.com | archivebox add --depth=1
archivebox list https://example.com | archivebox add --depth=1   # repeat up to desired depth
...
# all URLs that don't match *.example.com will be excluded, e.g. a link to youtube.com would not be followed)         Note: all assets required to render each page are still archived, URL_DENYLIST /URL_ALLOWLIST do not apply to images, css, video, etc. visible inline within the page.  Related options: URL_DENYLIST  , SAVE_MEDIA  , SAVE_GIT  , GIT_DOMAINS   Archive Method Toggles     High-level on/off switches for all the various methods used to archive URLs.  SAVE_TITLE      Possible Values: [True ]/False By default ArchiveBox uses the title provided by the import file, but not all types of imports provide titles (e.g. Plain texts lists of URLs).  When this is True, ArchiveBox downloads the page (and follows all redirects), then it attempts to parse the link's title from the first <title></title> tag found in the response.  It may be buggy or not work for certain sites that use JS to set the title, disabling it will lead to links imported without a title showing up with their URL as the title in the UI. Related options: ONLY_NEW  , CHECK_SSL_VALIDITY   SAVE_FAVICON      Possible Values: [True ]/False Fetch and save favicon for the URL from Google's public favicon service: https://www.google.com/s2/favicons?domain={domain} .  Set this to FALSE if you don't need favicons. Related options: TEMPLATES_DIR  , CHECK_SSL_VALIDITY  , CURL_BINARY   SAVE_WGET      Possible Values: [True ]/False Fetch page with wget, and save responses into folders for each domain, e.g. example.com/index.html , with .html appended if not present.  For a full list of options used during the wget download process, see the archivebox/archive_methods.py:save_wget(...) function. Related options: TIMEOUT  , SAVE_WGET_REQUISITES  , CHECK_SSL_VALIDITY  , COOKIES_FILE  , WGET_USER_AGENT  , SAVE_WARC  , WGET_BINARY   SAVE_WARC      Possible Values: [True ]/False Save a timestamped WARC archive of all the page requests and responses during the wget archive process. Related options: TIMEOUT  , SAVE_WGET_REQUISITES  , CHECK_SSL_VALIDITY  , COOKIES_FILE  , WGET_USER_AGENT  , SAVE_WGET  , WGET_BINARY   SAVE_PDF      Possible Values: [True ]/False Print page as PDF. Related options: TIMEOUT  , CHECK_SSL_VALIDITY  , CHROME_USER_DATA_DIR  , CHROME_BINARY   SAVE_SCREENSHOT      Possible Values: [True ]/False Fetch a screenshot of the page. Related options: RESOLUTION  , TIMEOUT  , CHECK_SSL_VALIDITY  , CHROME_USER_DATA_DIR  , CHROME_BINARY   SAVE_DOM      Possible Values: [True ]/False Fetch a DOM dump of the page. Related options: TIMEOUT  , CHECK_SSL_VALIDITY  , CHROME_USER_DATA_DIR  , CHROME_BINARY   SAVE_SINGLEFILE      Possible Values: [True ]/False Fetch an HTML file with all assets embedded using (https://github.com/gildas-lormeau/SingleFile) Single File . Related options: TIMEOUT  , CHECK_SSL_VALIDITY  , CHROME_USER_DATA_DIR  , CHROME_BINARY  , SINGLEFILE_BINARY   SAVE_READABILITY      Possible Values: [True ]/False Extract article text, summary, and byline using Mozilla's (https://github.com/mozilla/readability) Readability library.
Unlike the other methods, this does not download any additional files, so it's practically free from a disk usage perspective. It works by using any existing downloaded HTML version (e.g. wget, DOM dump, singlefile) and piping it into readability. Related options: TIMEOUT  , SAVE_WGET  , SAVE_DOM  , SAVE_SINGLEFILE  , SAVE_MERCURY   SAVE_MERCURY      Possible Values: [True ]/False Extract article text, summary, and byline using the (https://github.com/postlight/mercury-parser) Mercury library.
Unlike the other methods, this does not download any additional files, so it's practically free from a disk usage perspective. It works by using any existing downloaded HTML version (e.g. wget, DOM dump, singlefile) and piping it into Mercury. Related options: TIMEOUT  , SAVE_WGET  , SAVE_DOM  , SAVE_SINGLEFILE  , SAVE_READABILITY   SAVE_GIT      Possible Values: [True ]/False Fetch any git repositories on the page. Related options: TIMEOUT  , GIT_DOMAINS  , CHECK_SSL_VALIDITY  , GIT_BINARY   SAVE_MEDIA      Possible Values: [True ]/False Fetch all audio, video, annotations, and media metadata on the page using youtube-dl .  Warning, this can use up a lot of storage very quickly. Related options: MEDIA_TIMEOUT  , CHECK_SSL_VALIDITY  , YOUTUBEDL_BINARY   SAVE_ARCHIVE_DOT_ORG      Possible Values: [True ]/False Submit the page's URL to be archived on Archive.org. (The Internet Archive) Related options: TIMEOUT  , CHECK_SSL_VALIDITY  , CURL_BINARY   Archive Method Options     Specific options for individual archive methods above. Some of these are shared between multiple archive methods, others are specific to a single method.  CHECK_SSL_VALIDITY      Possible Values: [True ]/False Whether to enforce HTTPS certificate and HSTS chain of trust when archiving sites.  Set this to False if you want to archive pages even if they have expired or invalid certificates.  Be aware that when False you cannot guarantee that you have not been man-in-the-middle'd while archiving content, so the content cannot be verified to be what's on the original site. SAVE_WGET_REQUISITES      Possible Values: [True ]/False Fetch images/css/js with wget. (True is highly recommended, otherwise your won't download many critical assets to render the page, like images, js, css, etc.) Related options: TIMEOUT  , SAVE_WGET  , SAVE_WARC  , WGET_BINARY   RESOLUTION      Possible Values: [1440,2000 ]/1024,768 /...Screenshot resolution in pixels width,height. Related options: SAVE_SCREENSHOT   CURL_USER_AGENT      Possible Values: [Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.61 Safari/537.36 ArchiveBox/{VERSION} (+https://github.com/ArchiveBox/ArchiveBox/) curl/{CURL_VERSION} ]/"Mozilla/5.0 ..." /...This is the user agent to use during curl archiving.  You can set this to impersonate a more common browser like Chrome or Firefox if you're getting blocked by servers for having an unknown/blacklisted user agent. Related options: USE_CURL  , SAVE_TITLE  , CHECK_SSL_VALIDITY  , CURL_BINARY  , WGET_USER_AGENT  , CHROME_USER_AGENT   WGET_USER_AGENT      Possible Values: [Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.61 Safari/537.36 ArchiveBox/{VERSION} (+https://github.com/ArchiveBox/ArchiveBox/) wget/{WGET_VERSION} ]/"Mozilla/5.0 ..." /...This is the user agent to use during wget archiving.  You can set this to impersonate a more common browser like Chrome or Firefox if you're getting blocked by servers for having an unknown/blacklisted user agent. Related options: SAVE_WGET  , SAVE_WARC  , CHECK_SSL_VALIDITY  , WGET_BINARY  , CHROME_USER_AGENT   CHROME_USER_AGENT      Possible Values: [Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.61 Safari/537.36 ArchiveBox/{VERSION} (+https://github.com/ArchiveBox/ArchiveBox/) ]/"Mozilla/5.0 ..." /... This is the user agent to use during Chrome headless archiving.  If you're experiencing being blocked by many sites, you can set this to hide the Headless string that reveals to servers that you're using a headless browser. Related options: SAVE_PDF  , SAVE_SCREENSHOT  , SAVE_DOM  , CHECK_SSL_VALIDITY  , CHROME_USER_DATA_DIR  , CHROME_HEADLESS  , CHROME_BINARY  , WGET_USER_AGENT   GIT_DOMAINS      Possible Values: [github.com,bitbucket.org,gitlab.com,gist.github.com,codeberg.org,gitea.com,git.sr.ht ]/git.example.com /...Domains to attempt download of git repositories on using git clone . Related options: SAVE_GIT  , CHECK_SSL_VALIDITY   COOKIES_FILE      Possible Values: [None ]//path/to/cookies.txt /... Cookies file to pass to wget , curl , yt-dlp and other extractors that don't use Chrome (with its CHROME_USER_DATA_DIR ) for authentication.  To capture sites that require a user to be logged in, you configure this option to point to a (http://www.cookiecentral.com/faq/#3.5) netscape-format cookies.txt  file containing all the cookies you want to use during archiving. You can generate this cookies.txt file by using a number of different (https://chromewebstore.google.com/detail/get-cookiestxt-locally/cclelndahbckbenkjhflpdbgdldlbecc) browser extensions that can export your cookies in this format, or by using wget on the command line with --save-cookies + --user=... --password=... .   Warning Make sure you use separate burner credentials dedicated to archiving, e.g. don't re-use your normal daily Facebook/Instagram/Youtube/etc. account cookies as server responses often contain your name/email/PII, session tokens, etc. which then get preserved in your snapshots! Future viewers of your archive may be able to use any reflected (https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#%EF%B8%8F-things-to-watch-out-for-%EF%B8%8F) archived session tokens to log in as you, or at the very least, associate the content with your real identity. Even if this tradeoff seems acceptable now or you plan to keep your archive data private, you may want to share a snapshot with others in the future, and snapshots are very hard to sanitize/anonymize after-the-fact!  Related options: SAVE_WGET  , SAVE_WARC  , CHECK_SSL_VALIDITY  , WGET_BINARY   CHROME_USER_DATA_DIR      Possible Values: [~/.config/google-chrome ]//tmp/chrome-profile /... Path to a (https://chromium.googlesource.com/chromium/src/+/HEAD/docs/user_data_dir.md) Chrome user profile directory .  To capture sites that require a user to be logged in, you can specify a path to a Chrome user profile (which loads the cookies needed for the user to be logged in).  If you don't have an existing Chrome profile, create one with chromium-browser --user-data-dir=/tmp/chrome-profile , and log into the sites you need.  Then set CHROME_USER_DATA_DIR=/tmp/chrome-profile to make ArchiveBox use that profile. For a guide on how to set this up, see our (https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install#setting-up-a-chromium-user-profile) Chromium Install: Setting up a profile wiki. Note: Make sure the path does not have Default at the end (it should the the parent folder of Default ), e.g. set it to CHROME_USER_DATA_DIR=~/.config/chromium and not CHROME_USER_DATA_DIR=~/.config/chromium/Default .    Warning Make sure you use separate burner credentials dedicated to archiving, e.g. don't log in with your normal daily Facebook/Instagram/Youtube/etc. accounts as server responses and page content will often contain your name/email/PII, session cookies, private tokens, etc. which then get preserved in your snapshots! Future viewers of your archive may be able to use any reflected (https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#%EF%B8%8F-things-to-watch-out-for-%EF%B8%8F) archived session tokens to log in as you, or at the very least, associate the content with your real identity. Even if this tradeoff seems acceptable now or you plan to keep your archive data private, you may want to share a snapshot with others in the future, and snapshots are very hard to sanitize/anonymize after-the-fact!  When set to None , ArchiveBox <v0.7.2 used to try to find any existing profile on your system automatically, but this behavior has been disabled in later versions for security reasons, it must now be set explicitly if you want to use a profile. Related options: SAVE_PDF  , SAVE_SCREENSHOT  , SAVE_DOM  , CHECK_SSL_VALIDITY  , CHROME_HEADLESS  , CHROME_BINARY  , COOKIES_FILE   CHROME_HEADLESS      Possible Values: [True ]/False Whether or not to use Chrome/Chromium in --headless mode (no browser UI displayed).  When set to False , the full Chrome UI will be launched each time it's used to archive a page, which greatly slows down the process but allows you to watch in real-time as it saves each page. Related options: SAVE_PDF  , SAVE_SCREENSHOT  , SAVE_DOM  , CHROME_USER_DATA_DIR  , CHROME_BINARY   CHROME_SANDBOX      Possible Values: [True ]/False Whether or not to use the Chrome sandbox when archiving. If you see an error message like this, it means you are trying to run ArchiveBox as root: :ERROR:zygote_host_impl_linux.cc(89)] Running as root without --no-sandbox is not supported. See https://crbug.com/638180 (:ERROR:zygote_host_impl_linux.cc(89)] Running as root without --no-sandbox is not supported. See https://crbug.com/638180)         *Note: Do not run ArchiveBox as root! The solution to this error is not to override it by setting CHROME_SANDBOX=False , it's to use create another user (e.g. www-data ) and run ArchiveBox under that new, less privileged user. This is a security-critical setting, only set this to False if you're running ArchiveBox inside a container or VM where it doesn't have access to the rest of your system! Related options: SAVE_PDF  , SAVE_SCREENSHOT  , SAVE_DOM  , CHECK_SSL_VALIDITY  , CHROME_USER_DATA_DIR  , CHROME_HEADLESS  , CHROME_BINARY   Shell Options     Options around the format of the CLI output.  USE_COLOR      Possible Values: [True ]/False Colorize console output. Defaults to True if stdin is a TTY (interactive session), otherwise False (e.g. if run in a script or piped into a file). SHOW_PROGRESS      Possible Values: [True ]/False Show real-time progress bar in console output. Defaults to True if stdin is a TTY (interactive session), otherwise False (e.g. if run in a script or piped into a file). Note: We use (https://gist.github.com/pirate/c89b7d42be148e9180d8c7cf81e734c8) asymptotic progress bars because most tasks complete quickly! ✨  Dependency Options     Options for defining which binaries to use for the various archive method dependencies.  CHROME_BINARY      Possible Values: [chromium-browser ]//usr/local/bin/google-chrome /...Path or name of the Google Chrome / Chromium binary to use for all the headless browser archive methods. Without setting this environment variable, ArchiveBox by default look for the following binaries in $PATH in this order: chromium-browser  chromium  google-chrome  google-chrome-stable  google-chrome-unstable  google-chrome-beta  google-chrome-canary  google-chrome-dev   You can override the default behavior to search for any available bin by setting the environment variable to your preferred Chrome binary name or path. The chrome/chromium dependency is optional and only required for screenshots, PDF, and DOM dump output, it can be safely ignored if those three methods are disabled. Related options: SAVE_PDF  , SAVE_SCREENSHOT  , SAVE_DOM  , SAVE_SINGLEFILE  , CHROME_USER_DATA_DIR  , CHROME_HEADLESS  , CHROME_SANDBOX   WGET_BINARY      Possible Values: [wget ]//usr/local/bin/wget /...Path or name of the wget binary to use. Related options: SAVE_WGET  , SAVE_WARC   YOUTUBEDL_BINARY      Possible Values: [youtube-dl ]//usr/local/bin/youtube-dl /...Path or name of the (https://github.com/rg3/youtube-dl) youtube-dl binary to use. Related options: SAVE_MEDIA   GIT_BINARY      Possible Values: [git ]//usr/local/bin/git /...Path or name of the git binary to use. Related options: SAVE_GIT   CURL_BINARY      Possible Values: [curl ]//usr/local/bin/curl /...Path or name of the curl binary to use. Related options: SAVE_FAVICON  , SAVE_ARCHIVE_DOT_ORG   SINGLEFILE_BINARY      Possible Values: [single-file ]/./node_modules/single-file/cli/single-file /...Path or name of the SingleFile binary to use. This can be installed using npm install --no-audit --no-fund 'git+https://github.com/gildas-lormeau/SingleFile.git' . Related options: SAVE_SINGLEFILE  , CHROME_BINARY  , CHROME_USER_DATA_DIR  , CHROME_HEADLESS  , CHROME_SANDBOX   READABILITY_BINARY      Possible Values: [readability-extractor ]/./node_modules/readability-extractor/readability-extractor /...Path or name of the Readability extrator binary to use. This can be installed using npm install --no-audit --no-fund 'git+https://github.com/ArchiveBox/readability-extractor.git' . Related options: SAVE_READABILITY   MERCURY_BINARY      Possible Values: [mercury-parser ]/./node_modules/@postlight/mercury-parser/cli.js /...Path or name of the Mercury parser extractor binary to use. This can be installed using npm install --no-audit --no-fund '@postlight/mercury-parser' . Related options: SAVE_MERCURY   RIPGREP_BINARY      Possible Values: [rg ]/rga /... Path or name of the ripgrep binary to use for full text search. This can be installed using your system package manager, e.g. apt install ripgrep or brew install ripgrep . Optionally switch this to use ripgrep-all for full-text search support across more filetypes (e.g. PDF): (https://github.com/phiresky/ripgrep-all) https://github.com/phiresky/ripgrep-all . Related options: SEARCH_BACKEND_ENGINE   SINGLEFILE_ARGS      Possible Values: [["--back-end=playwright-firefox","--load-deferred-images-dispatch-scroll-event=true"] ]/.. Arguments that are passed to the SingleFile binary. The values should be a valid JSON string. Related options: SINGLEFILE_BINARY   CURL_ARGS      Possible Values: [["--tlsv1.3","--http2"] ]/.. Arguments that are passed to the curl binary. The values should be a valid JSON string. Related options: CURL_BINARY   WGET_ARGS      Possible Values: [["--https-only"] ]/.. Arguments that are passed to the wget binary. The values should be a valid JSON string. Related options: WGET_BINARY   YOUTUBEDL_ARGS      Possible Values: [["--limit-rate=10M"] ]/.. Arguments that are passed to the (https://github.com/rg3/youtube-dl) youtube-dl binary. The values should be a valid JSON string. Related options: YOUTUBEDL_BINARY   GIT_ARGS      Possible Values: [["--depth=1"] ] Arguments that are passed to the git clone subcommand. The values should be a valid JSON string. Related options: GIT_BINARY   ▶️ Looking for more? Sometimes this document is out of date. Check the source code for extra undocumented options: (https://github.com/ArchiveBox/ArchiveBox/blob/master/archivebox/config.py#L27) archivebox/config.py  .   (https://github.com/ArchiveBox/ArchiveBox/issues/new?assignees=&labels=&template=documentation_change.md&title=) ✏️ Help improve our documentation...            Toggle table of contents Pages (37) 37       Loading    (https://github.com/ArchiveBox/ArchiveBox/wiki) Home    (https://github.com/ArchiveBox/ArchiveBox/wiki#archivebox-documentation) ArchiveBox Documentation        Loading    (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox) archivebox    (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox#archivebox-package) archivebox package  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox#subpackages) Subpackages  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox#submodules) Submodules  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox#archiveboxmain-module) archivebox.main module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox#archiveboxmanage-module) archivebox.manage module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox#archiveboxsystem-module) archivebox.system module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox#archiveboxutil-module) archivebox.util module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox#module-contents) Module contents        Loading    (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.cli) archivebox.cli    (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.cli#archiveboxcli-package) archivebox.cli package  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.cli#submodules) Submodules  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.cli#archiveboxcliarchivebox_add-module) archivebox.cli.archivebox_add module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.cli#archiveboxcliarchivebox_config-module) archivebox.cli.archivebox_config module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.cli#archiveboxcliarchivebox_help-module) archivebox.cli.archivebox_help module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.cli#archiveboxcliarchivebox_status-module) archivebox.cli.archivebox_status module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.cli#archiveboxcliarchivebox_init-module) archivebox.cli.archivebox_init module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.cli#archiveboxcliarchivebox_list-module) archivebox.cli.archivebox_list module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.cli#archiveboxcliarchivebox_manage-module) archivebox.cli.archivebox_manage module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.cli#archiveboxcliarchivebox_remove-module) archivebox.cli.archivebox_remove module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.cli#archiveboxcliarchivebox_schedule-module) archivebox.cli.archivebox_schedule module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.cli#archiveboxcliarchivebox_server-module) archivebox.cli.archivebox_server module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.cli#archiveboxcliarchivebox_shell-module) archivebox.cli.archivebox_shell module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.cli#archiveboxcliarchivebox_update-module) archivebox.cli.archivebox_update module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.cli#archiveboxcliarchivebox_version-module) archivebox.cli.archivebox_version module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.cli#module-contents) Module contents        Loading    (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.config) archivebox.config    (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.config#archiveboxconfig-package) archivebox.config package  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.config#submodules) Submodules  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.config#module-contents) Module contents  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.config#archiveboxconfigstubs-module) archivebox.config.stubs module        Loading    (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.core) archivebox.core    (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.core#archiveboxcore-package) archivebox.core package  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.core#subpackages) Subpackages  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.core#submodules) Submodules  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.core#archiveboxcoreadmin-module) archivebox.core.admin module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.core#archiveboxcoreapps-module) archivebox.core.apps module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.core#archiveboxcoresettings-module) archivebox.core.settings module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.core#archiveboxcoretests-module) archivebox.core.tests module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.core#archiveboxcoreurls-module) archivebox.core.urls module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.core#archiveboxcoreviews-module) archivebox.core.views module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.core#archiveboxcorewelcome_message-module) archivebox.core.welcome_message module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.core#archiveboxcorewsgi-module) archivebox.core.wsgi module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.core#module-contents) Module contents        Loading    (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.core.migrations) archivebox.core.migrations    (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.core.migrations#archiveboxcoremigrations-package) archivebox.core.migrations package  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.core.migrations#submodules) Submodules  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.core.migrations#archiveboxcoremigrations0001_initial-module) archivebox.core.migrations.0001_initial module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.core.migrations#module-contents) Module contents        Loading    (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.core.templatetags) archivebox.core.templatetags    (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.core.templatetags#archiveboxcoretemplatetags-package) archivebox.core.templatetags package  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.core.templatetags#submodules) Submodules  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.core.templatetags#archiveboxcoretemplatetagscore_tags-module) archivebox.core.templatetags.core_tags module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.core.templatetags#module-contents) Module contents        Loading    (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.extractors) archivebox.extractors    (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.extractors#archiveboxextractors-package) archivebox.extractors package  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.extractors#submodules) Submodules  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.extractors#archiveboxextractorsarchive_org-module) archivebox.extractors.archive_org module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.extractors#archiveboxextractorsdom-module) archivebox.extractors.dom module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.extractors#archiveboxextractorsfavicon-module) archivebox.extractors.favicon module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.extractors#archiveboxextractorsgit-module) archivebox.extractors.git module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.extractors#archiveboxextractorsmedia-module) archivebox.extractors.media module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.extractors#archiveboxextractorspdf-module) archivebox.extractors.pdf module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.extractors#archiveboxextractorsscreenshot-module) archivebox.extractors.screenshot module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.extractors#archiveboxextractorstitle-module) archivebox.extractors.title module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.extractors#archiveboxextractorswget-module) archivebox.extractors.wget module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.extractors#module-contents) Module contents        Loading    (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.index) archivebox.index    (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.index#archiveboxindex-package) archivebox.index package  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.index#submodules) Submodules  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.index#archiveboxindexcsv-module) archivebox.index.csv module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.index#archiveboxindexhtml-module) archivebox.index.html module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.index#archiveboxindexjson-module) archivebox.index.json module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.index#archiveboxindexschema-module) archivebox.index.schema module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.index#archiveboxindexsql-module) archivebox.index.sql module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.index#module-contents) Module contents        Loading    (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.parsers) archivebox.parsers    (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.parsers#archiveboxparsers-package) archivebox.parsers package  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.parsers#submodules) Submodules  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.parsers#archiveboxparsersgeneric_json-module) archivebox.parsers.generic_json module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.parsers#archiveboxparsersgeneric_rss-module) archivebox.parsers.generic_rss module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.parsers#archiveboxparsersgeneric_txt-module) archivebox.parsers.generic_txt module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.parsers#archiveboxparsersmedium_rss-module) archivebox.parsers.medium_rss module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.parsers#archiveboxparsersnetscape_html-module) archivebox.parsers.netscape_html module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.parsers#archiveboxparserspinboard_rss-module) archivebox.parsers.pinboard_rss module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.parsers#archiveboxparserspocket_html-module) archivebox.parsers.pocket_html module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.parsers#archiveboxparsersshaarli_rss-module) archivebox.parsers.shaarli_rss module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.parsers#module-contents) Module contents        Loading    (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.search) archivebox.search    (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.search#archiveboxsearch-package) archivebox.search package  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.search#subpackages) Subpackages  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.search#submodules) Submodules  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.search#archiveboxsearchutils-module) archivebox.search.utils module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.search#module-contents) Module contents        Loading    (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.search.backends) archivebox.search.backends    (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.search.backends#archiveboxsearchbackends-package) archivebox.search.backends package  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.search.backends#submodules) Submodules  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.search.backends#archiveboxsearchbackendsripgrep-module) archivebox.search.backends.ripgrep module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.search.backends#archiveboxsearchbackendssonic-module) archivebox.search.backends.sonic module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.search.backends#module-contents) Module contents        Loading    (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.vendor) archivebox.vendor    (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.vendor#archiveboxvendor-package) archivebox.vendor package  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.vendor#submodules) Submodules  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.vendor#archiveboxvendorbase32_crockford-module) archivebox.vendor.base32_crockford module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.vendor#archiveboxvendorpocket-module) archivebox.vendor.pocket module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.vendor#archiveboxvendortaggit_utils-module) archivebox.vendor.taggit_utils module  (https://github.com/ArchiveBox/ArchiveBox/wiki/archivebox.vendor#module-contents) Module contents        Loading    (https://github.com/ArchiveBox/ArchiveBox/wiki/Changelog) Changelog    (https://github.com/ArchiveBox/ArchiveBox/wiki/Changelog#changelog) Changelog        Loading    (https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install) Chromium Install    (https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install#chromium-install) Chromium Install  (https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install#installing-chromium) Installing Chromium  (https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install#%EF%B8%8F-any-os-recommended) ⭐️ Any OS (recommended)  (https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install#macos) macOS  (https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install#ubuntudebian) Ubuntu/Debian  (https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install#installing-google-chrome) Installing Google Chrome  (https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install#macos-1) macOS  (https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install#ubuntudebian-1) Ubuntu/Debian  (https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install#troubleshooting-chromium-install) Troubleshooting Chromium Install  (https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install#setting-up-a-chromium-user-profile) Setting Up a Chromium User Profile  (https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install#docker-vnc-setup) Docker VNC Setup  (https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install#non-docker-setup-local-host) Non-Docker Setup (Local Host)  (https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install#non-docker-setup-remote-host) Non-Docker Setup (Remote Host)  (https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install#more-info--troubleshooting) More Info & Troubleshooting        Loading    (https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration) Configuration    Configuration  General Settings  OUTPUT_PERMISSIONS  PUID / PGID  ONLY_NEW  TIMEOUT  MEDIA_TIMEOUT  ADMIN_USERNAME / ADMIN_PASSWORD  PUBLIC_INDEX / PUBLIC_SNAPSHOTS / PUBLIC_ADD_VIEW  CUSTOM_TEMPLATES_DIR  REVERSE_PROXY_USER_HEADER  REVERSE_PROXY_WHITELIST  LOGOUT_REDIRECT_URL  LDAP  SNAPSHOTS_PER_PAGE  FOOTER_INFO  URL_DENYLIST  URL_ALLOWLIST  Archive Method Toggles  SAVE_TITLE  SAVE_FAVICON  SAVE_WGET  SAVE_WARC  SAVE_PDF  SAVE_SCREENSHOT  SAVE_DOM  SAVE_SINGLEFILE  SAVE_READABILITY  SAVE_MERCURY  SAVE_GIT  SAVE_MEDIA  SAVE_ARCHIVE_DOT_ORG  Archive Method Options  CHECK_SSL_VALIDITY  SAVE_WGET_REQUISITES  RESOLUTION  CURL_USER_AGENT  WGET_USER_AGENT  CHROME_USER_AGENT  GIT_DOMAINS  COOKIES_FILE  CHROME_USER_DATA_DIR  CHROME_HEADLESS  CHROME_SANDBOX  Shell Options  USE_COLOR  SHOW_PROGRESS  Dependency Options  CHROME_BINARY  WGET_BINARY  YOUTUBEDL_BINARY  GIT_BINARY  CURL_BINARY  SINGLEFILE_BINARY  READABILITY_BINARY  MERCURY_BINARY  RIPGREP_BINARY  SINGLEFILE_ARGS  CURL_ARGS  WGET_ARGS  YOUTUBEDL_ARGS  GIT_ARGS        Loading    (https://github.com/ArchiveBox/ArchiveBox/wiki/Contents) Contents    (https://github.com/ArchiveBox/ArchiveBox/wiki/Contents#intro) Intro  (https://github.com/ArchiveBox/ArchiveBox/wiki/Contents#getting-started) Getting Started  (https://github.com/ArchiveBox/ArchiveBox/wiki/Contents#general) General  (https://github.com/ArchiveBox/ArchiveBox/wiki/Contents#api-reference) API Reference  (https://github.com/ArchiveBox/ArchiveBox/wiki/Contents#meta) Meta        Loading    (https://github.com/ArchiveBox/ArchiveBox/wiki/Docker) Docker    (https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#docker) Docker  (https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#overview) Overview  (https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#docker-compose) Docker Compose  (https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#setup) Setup  (https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#upgrading) Upgrading  (https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#usage) Usage  (https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#accessing-the-data) Accessing the data  (https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#configuration) Configuration  (https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#docker-1) Docker  (https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#setup-1) Setup  (https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#upgrading-1) Upgrading  (https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#usage-1) Usage  (https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#accessing-the-data-1) Accessing the data  (https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#configuration-1) Configuration        Loading    (https://github.com/ArchiveBox/ArchiveBox/wiki/Donations) Donations    (https://github.com/ArchiveBox/ArchiveBox/wiki/Donations#supporting-development) Supporting Development        Loading    (https://github.com/ArchiveBox/ArchiveBox/wiki/index) index    (https://github.com/ArchiveBox/ArchiveBox/wiki/index#archivebox) ArchiveBox  (https://github.com/ArchiveBox/ArchiveBox/wiki/index#documentation) Documentation        Loading    (https://github.com/ArchiveBox/ArchiveBox/wiki/Install) Install    (https://github.com/ArchiveBox/ArchiveBox/wiki/Install#install) Install  (https://github.com/ArchiveBox/ArchiveBox/wiki/Install#supported-systems) Supported Systems  (https://github.com/ArchiveBox/ArchiveBox/wiki/Install#option-a-docker--docker-compose-setup-%EF%B8%8F) Option A. Docker / Docker Compose Setup ⭐️  (https://github.com/ArchiveBox/ArchiveBox/wiki/Install#option-b-automatic-setup-script) Option B. Automatic Setup Script  (https://github.com/ArchiveBox/ArchiveBox/wiki/Install#option-c-bare-metal-setup) Option C. Bare Metal Setup  (https://github.com/ArchiveBox/ArchiveBox/wiki/Install#1-install-base-system-dependencies-needed-for-your-os) 1. Install base system dependencies needed for your OS  (https://github.com/ArchiveBox/ArchiveBox/wiki/Install#macos) macOS  (https://github.com/ArchiveBox/ArchiveBox/wiki/Install#ubuntudebian-based-systems) Ubuntu/Debian-based Systems  (https://github.com/ArchiveBox/ArchiveBox/wiki/Install#freebsd) FreeBSD  (https://github.com/ArchiveBox/ArchiveBox/wiki/Install#openbsd) OpenBSD  (https://github.com/ArchiveBox/ArchiveBox/wiki/Install#arch-linux--nix--guix--etc-other-oss) Arch Linux / Nix / Guix / etc. Other OSs  (https://github.com/ArchiveBox/ArchiveBox/wiki/Install#2-install-the-python-dependencies-using-pip) 2. Install the Python dependencies using pip  (https://github.com/ArchiveBox/ArchiveBox/wiki/Install#3-install-the-js-dependencies-using-archivebox-init---setup) 3. Install the JS dependencies using archivebox init --setup  (https://github.com/ArchiveBox/ArchiveBox/wiki/Install#troubleshooting) Troubleshooting  (https://github.com/ArchiveBox/ArchiveBox/wiki/Install#next-steps-add-some-urls-to-archive-and-try-out-cli--web-ui) Next Steps: Add some URLs to archive and try out CLI / Web UI  (https://github.com/ArchiveBox/ArchiveBox/wiki/Install#next-steps-upgrading-archivebox-to-a-new-version) Next Steps: Upgrading Archivebox to a new version  (https://github.com/ArchiveBox/ArchiveBox/wiki/Install#further-reading) Further Reading        Loading    (https://github.com/ArchiveBox/ArchiveBox/wiki/Merging-Collections) Merging Collections    (https://github.com/ArchiveBox/ArchiveBox/wiki/Merging-Collections#merging-two-or-more-existing-archives) Merging two or more existing archives  (https://github.com/ArchiveBox/ArchiveBox/wiki/Merging-Collections#modify-the-archivebox-sqlite3-db-directly) Modify the ArchiveBox SQLite3 DB directly  (https://github.com/ArchiveBox/ArchiveBox/wiki/Merging-Collections#example-modifying-an-existing-users-email) Example: Modifying an existing user's email  (https://github.com/ArchiveBox/ArchiveBox/wiki/Merging-Collections#example-adding-a-new-user-with-a-hashed-password) Example: Adding a new user with a hashed password  (https://github.com/ArchiveBox/ArchiveBox/wiki/Merging-Collections#database-troubleshooting) Database Troubleshooting  (https://github.com/ArchiveBox/ArchiveBox/wiki/Merging-Collections#related-documents) Related Documents        Loading    (https://github.com/ArchiveBox/ArchiveBox/wiki/modules) modules    (https://github.com/ArchiveBox/ArchiveBox/wiki/modules#archivebox) archivebox        Loading    (https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive) Publishing Your Archive    (https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive#publishing-your-archive) Publishing Your Archive  (https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive#1-use-the-built-in-web-server) 1. Use the built-in web server  (https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive#2-export-and-host-it-as-static-html) 2. Export and host it as static HTML  (https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive#security-concerns) Security Concerns  (https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive#protecting-the-admin-dashboard) Protecting the Admin Dashboard  (https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive#copyright-concerns) Copyright Concerns  (https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive#further-reading-usa-copyright-law--fair-use-exemptions) Further Reading: USA Copyright Law & Fair Use Exemptions        Loading    (https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart) Quickstart    (https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart#quickstart) Quickstart  (https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart#1-set-up-archivebox) 1. Set up ArchiveBox  (https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart#2-get-your-list-of-urls-to-archive) 2. Get your list of URLs to archive  (https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart#3-add-your-urls-to-the-archive) 3. Add your URLs to the archive  (https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart#-done) ✅ Done!        Loading    (https://github.com/ArchiveBox/ArchiveBox/wiki/README) README         Loading    (https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap) Roadmap    (https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap#roadmap) Roadmap  (https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap#planned-specification) Planned Specification  (https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap#v07-schema-improvements) v0.7: Schema improvements  (https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap#v08--security) v0.8:  Security  (https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap#v09--performance) v0.9:  Performance  (https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap#v10-full-headless-browser-control) v1.0: Full headless browser control  (https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap#v20-federated-or-distributed-archiving--paid-hosted-service-offering) v2.0 Federated or distributed archiving + paid hosted service offering  (https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap#major-long-term-changes) Major long-term changes  (https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap#smaller-planned-features) Smaller planned features  (https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap#past-releases) Past Releases  (https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap#ui--ux-improvements-planned) UI / UX Improvements Planned  (https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap#new-extractors-planned) New Extractors Planned  (https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap#social-media) Social Media  (https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap#videostreams) Video/Streams  (https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap#audiomusic) Audio/Music  (https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap#photosimagescomics) Photos/Images/Comics  (https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap#textforums) Text/Forums  (https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap#mooceducational-content) MOOC/Educational Content  (https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap#re-archiving--warc-creation) Re-Archiving / WARC Creation  (https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap#other) Other        Loading    (https://github.com/ArchiveBox/ArchiveBox/wiki/Scheduled-Archiving) Scheduled Archiving    (https://github.com/ArchiveBox/ArchiveBox/wiki/Scheduled-Archiving#scheduled-archiving) Scheduled Archiving  (https://github.com/ArchiveBox/ArchiveBox/wiki/Scheduled-Archiving#docker-usage) Docker Usage  (https://github.com/ArchiveBox/ArchiveBox/wiki/Scheduled-Archiving#example-archive-a-twitter-users-tweets-and-linked-content-within-once-a-week) Example: Archive a Twitter user's Tweets and linked content within once a week  (https://github.com/ArchiveBox/ArchiveBox/wiki/Scheduled-Archiving#example-archive-a-reddit-subreddit-and-discussions-for-every-post-once-a-week) Example: Archive a Reddit subreddit and discussions for every post once a week  (https://github.com/ArchiveBox/ArchiveBox/wiki/Scheduled-Archiving#example-archive-the-hackernews-front-page-and-some-linked-articles-every-24-hours) Example: Archive the HackerNews front page and some linked articles every 24 hours  (https://github.com/ArchiveBox/ArchiveBox/wiki/Scheduled-Archiving#example-archive-all-urls-in-an-rss-feed-from-pocket-every-12-hours) Example: Archive all URLs in an RSS feed from Pocket every 12 hours  (https://github.com/ArchiveBox/ArchiveBox/wiki/Scheduled-Archiving#example-archive-a-github-repositorys-source-code-only-once-a-month) Example: Archive a Github repository's source code only once a month  (https://github.com/ArchiveBox/ArchiveBox/wiki/Scheduled-Archiving#example-archive-a-list-of-urls-pulled-from-the-filesystem-every-30-minutes) Example: Archive a list of URLs pulled from the filesystem every 30 minutes  (https://github.com/ArchiveBox/ArchiveBox/wiki/Scheduled-Archiving#advanced-scheduling-using-cron) Advanced Scheduling Using Cron  (https://github.com/ArchiveBox/ArchiveBox/wiki/Scheduled-Archiving#example-export-and-archive-firefox-browser-history-every-24-hours) Example: Export and archive Firefox browser history every 24 hours  (https://github.com/ArchiveBox/ArchiveBox/wiki/Scheduled-Archiving#example-import-an-rss-feed-from-pocket-every-12-hours) Example: Import an RSS feed from Pocket every 12 hours        Loading    (https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview) Security Overview    (https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#web-ui-permissions) Web UI Permissions  (https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#archivebox-use-cases) ArchiveBox Use-Cases  (https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#archiving-public-content-only-%EF%B8%8F-default-recommended-for-most-people) Archiving Public Content Only ⭐️ [Default, recommended for most people]  (https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#archiving-content-behind-log-ins--advanced-users-only) Archiving Content Behind Log-Ins 🚨 [Advanced users only]  (https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#%EF%B8%8F-things-to-watch-out-for-%EF%B8%8F) ⚠️ Things to watch out for: ⚠️  (https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#publishing) Publishing  (https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#do-not-run-as-root) Do not run as root  (https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#output-folder) Output Folder  (https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#database) Database  (https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#filesystem) Filesystem  (https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#purging-entries) Purging entries  (https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#permissions) Permissions        Loading    (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Authentication) Setting up Authentication    (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Authentication#set-up-admin-web-ui-permissions) Set Up Admin Web UI Permissions  (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Authentication#admin-web-ui-authentication-methods) Admin Web UI Authentication Methods  (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Authentication#username--password-the-default) Username & Password (the default)  (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Authentication#reverse-proxy-authentication) Reverse Proxy Authentication  (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Authentication#ldap-authentication) LDAP Authentication  (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Authentication#not-yet-supported-saml--oauth2--openid-authentication) Not Yet Supported: SAML / OAuth2 / OpenID Authentication  (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Authentication#rest-api) REST API  (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Authentication#api-bearer-token-authentication) API Bearer Token Authentication  (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Authentication#api-request-header-authentication) API Request Header Authentication  (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Authentication#api-query-parameter-authentication) API Query Parameter Authentication  (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Authentication#api-session-cookie-authentication) API Session Cookie Authentication  (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Authentication#api-http-basic-authentication) API HTTP Basic Authentication  (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Authentication#further-reading) Further Reading        Loading    (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Search) Setting up Search    (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Search#how-to-search-in-archivebox) How to Search in ArchiveBox  (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Search#how-search-works) How Search Works  (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Search#archivebox-search-backends) ArchiveBox Search Backends  (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Search#ripgrep-the-default) ripgrep(the default)  (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Search#pros) Pros  (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Search#cons) Cons  (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Search#ripgrep-all-aka-rga) ripgrep-all (aka rga)  (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Search#ugrep) ugrep  (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Search#pros-1) Pros  (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Search#cons-1) Cons  (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Search#sonic-%EF%B8%8F-the-recommended-upgrade-path-for-most-people) sonic ⭐️ (the recommended upgrade path for most people)  (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Search#pros-2) Pros  (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Search#cons-2) Cons  (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Search#sqlite-fts5) SQLite FTS5  (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Search#pros-3) Pros  (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Search#cons-3) Cons  (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Search#further-reading) Further Reading        Loading    (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-Up-Storage) Setting Up Storage    (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-Up-Storage#related-docs) Related Docs  (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-Up-Storage#supported-local-filesystems) Supported Local Filesystems  (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-Up-Storage#ext4-default-on-linux-apfs-default-on-macos) EXT4 (default on Linux), APFS (default on macOS)  (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-Up-Storage#zfs-recommended-for-best-experience-on-linuxbsd-%EF%B8%8F) ZFS (recommended for best experience on Linux/BSD) ⭐️  (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-Up-Storage#ntfs-hfs-btrfs) NTFS, HFS+, BTRFS  (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-Up-Storage#ext2-ext3-fat32-exfat) EXT2, EXT3, FAT32, exFAT  (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-Up-Storage#supported-remote-filesystems) Supported Remote Filesystems  (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-Up-Storage#nfs-docker-driver) NFS (Docker Driver)  (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-Up-Storage#smb--ceph-docker-cifs-driver) SMB / Ceph (Docker CIFS Driver)  (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-Up-Storage#amazon-s3--backblaze-b2--google-drive--etc-rclone) Amazon S3 / Backblaze B2 / Google Drive / etc. (RClone)  (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-Up-Storage#rclone-config-examples) RClone Config Examples  (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-Up-Storage#option-a-running-rclone-on-bare-metal-host) Option A: Running RClone on Bare Metal host  (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-Up-Storage#option-b-running-rclone-with-docker-storage-plugin) Option B: Running RClone with Docker Storage Plugin  (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-Up-Storage#more-docker-storage-plugins) More Docker Storage Plugins        Loading    (https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting) Troubleshooting    (https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting#troubleshooting) Troubleshooting  (https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting#installing) Installing  (https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting#python) Python  (https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting#chromiumgoogle-chrome) Chromium/Google Chrome  (https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting#wget--curl) Wget & Curl  (https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting#npm-dependencies) NPM Dependencies  (https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting#archiving) Archiving  (https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting#no-links-parsed-from-export-file) No links parsed from export file  (https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting#lots-of-skipped-sites) Lots of skipped sites  (https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting#lots-of-errors) Lots of errors  (https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting#lots-of-broken-links-from-the-index) Lots of broken links from the index  (https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting#removing-unwanted-links-from-the-index) Removing unwanted links from the index  (https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting#hosting-the-archive) Hosting the Archive  (https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting#other-database-or-filesystem-issues) Other database or filesystem issues  (https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting#docker-permissions-issues) Docker Permissions issues  (https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting#database) Database  (https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting#filesystem-doesnt-support-fsync-eg-network-mounts) Filesystem doesn't support FSYNC (e.g. network mounts)  (https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting#database-and-filesystem-contention-issues-when-running-multiple-archivebox-processes) Database and filesystem contention issues when running multiple ArchiveBox processes  (https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting#database-migrations-errors-or-upgrade-issues) Database migrations errors or upgrade issues  (https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting#repairing-a-corrupted-sqlite3-database-file) Repairing a corrupted SQLite3 database file        Loading    (https://github.com/ArchiveBox/ArchiveBox/wiki/Upgrading) Upgrading    (https://github.com/ArchiveBox/ArchiveBox/wiki/Upgrading#upgrade-your-archivebox-collection-to-a-new-version) Upgrade your ArchiveBox collection to a new version  (https://github.com/ArchiveBox/ArchiveBox/wiki/Upgrading#upgrading-with-docker-compose-%EF%B8%8F) Upgrading with Docker Compose ⭐️  (https://github.com/ArchiveBox/ArchiveBox/wiki/Upgrading#upgrading-with-plain-docker) Upgrading with plain Docker  (https://github.com/ArchiveBox/ArchiveBox/wiki/Upgrading#upgrading-with-a-package-manager) Upgrading with a package manager  (https://github.com/ArchiveBox/ArchiveBox/wiki/Upgrading#merge-two-or-more-existing-archives) Merge two or more existing archives  (https://github.com/ArchiveBox/ArchiveBox/wiki/Upgrading#related-documents) Related Documents        Loading    (https://github.com/ArchiveBox/ArchiveBox/wiki/Upgrading-or-Merging-Archives) Upgrading or Merging Archives         Loading    (https://github.com/ArchiveBox/ArchiveBox/wiki/Usage) Usage    (https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#usage) Usage  (https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#cli-usage) CLI Usage  (https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#run-archivebox-with-configuration-options) Run ArchiveBox with configuration options  (https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#import-a-single-url) Import a single URL  (https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#import-a-list-of-urls-from-a-text-file) Import a list of URLs from a text file  (https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#import-list-of-links-from-browser-history) Import list of links from browser history  (https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#ui-usage) UI Usage  (https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#explanation-of-buttons-in-the-web-ui---admin-snapshots-list) Explanation of buttons in the web UI - admin snapshots list  (https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#browser-extension-usage) Browser Extension Usage  (https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#more-info) More Info  (https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#disk-layout) Disk Layout  (https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#large-archives) Large Archives  (https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#sql-shell-usage) SQL Shell Usage  (https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#python-shell-usage) Python Shell Usage  (https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#python-api-usage) Python API Usage        Loading    (https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community) Web Archiving Community    (https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#web-archiving-community) Web Archiving Community  (https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#the-master-lists) The Master Lists  (https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#web-archiving-projects) Web Archiving Projects  (https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#bookmarking-services) Bookmarking Services  (https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#from-the-archiveorg--archive-it-teams) From the Archive.org & Archive-It teams  (https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#from-the-webrecorderio-team-and-the-rhizomeorgconifer-team) From the WebRecorder.io team and the Rhizome.org/Conifer team  (https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#from-the-old-dominion-university-web-science-team) From the Old Dominion University: Web Science Team  (https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#from-the-archives-unleashed-team) From the Archives Unleashed Team  (https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#from-the-iipc-team) From the IIPC team  (https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#other-public-archiving-services) Other Public Archiving Services  (https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#other-archivebox-alternatives) Other ArchiveBox Alternatives  (https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#smaller-utilities) Smaller Utilities  (https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#reading-list) Reading List  (https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#blogs-friends-of-archivebox) Blogs Friends of ArchiveBox  (https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#articles-we-like-about-internet-archiving) Articles We Like About Internet Archiving  (https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#archivebox-specific-posts-tutorials-and-guides) ArchiveBox-Specific Posts, Tutorials, and Guides  (https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#archivebox-discussions-in-news--social-media) ArchiveBox Discussions in News & Social Media  (https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#communities) Communities  (https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#most-active-communities) Most Active Communities  (https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#web-archiving-communities) Web Archiving Communities  (https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#general-archiving-foundations-coalitions-initiatives-and-institutes) General Archiving Foundations, Coalitions, Initiatives, and Institutes  (https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#archivebox-community-resources) ArchiveBox Community Resources  (https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#archivebox-chat-rooms) ArchiveBox Chat Rooms  (https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#archivebox-on-social-media) ArchiveBox on Social Media  (https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#archivebox-on-package-distribution-platforms) ArchiveBox on Package Distribution Platforms         (https://github.com/ArchiveBox/ArchiveBox/wiki/Home)   (https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart) Getting Started      🔢 (https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart) Quickstart  🖥️ (https://github.com/ArchiveBox/ArchiveBox/wiki/Install) Install  🐳 (https://github.com/ArchiveBox/ArchiveBox/wiki/Docker) Docker  ➡️ (https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart#2-get-your-list-of-urls-to-archive) Supported Sources  ⬅️ (https://github.com/ArchiveBox/ArchiveBox#output-formats) Supported Outputs   (https://github.com/ArchiveBox/ArchiveBox/wiki/Usage) Usage      ﹩(https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#cli-usage) Command Line  🌐 (https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#ui-usage) Web UI  🧩 (https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#browser-extension-usage) Browser Extension  👾 (https://github.com/ArchiveBox/ArchiveBox/issues/496#issuecomment-2080174235) REST API / (https://github.com/ArchiveBox/ArchiveBox/pull/1418) Webhooks  📜 (https://docs.archivebox.io/en/latest/modules.html) Python API / (https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#python-shell-usage) REPL / (https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#sql-shell-usage) SQL API   Reference     ⚙️ (https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration) Configuration  📦 (https://github.com/ArchiveBox/ArchiveBox#dependencies) Dependencies  💿 (https://github.com/ArchiveBox/ArchiveBox#archive-layout) Disk Layout  🔒 (https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview) Security Overview  📝 (https://github.com/ArchiveBox/ArchiveBox#archivebox-development) Developer Documentation   Guides     (https://github.com/ArchiveBox/ArchiveBox/wiki/Upgrading) Upgrading  (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-Up-Storage) Setting up Storage (NFS/SMB/S3/etc)  (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Authentication) Setting up Authentication (SSO/LDAP/etc)  (https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Search) Setting up Search (rg/sonic/etc)  (https://github.com/ArchiveBox/ArchiveBox/wiki/Scheduled-Archiving) Scheduled Archiving  (https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive) Publishing Your Archive  (https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install) Chromium Install  (https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install#setting-up-a-chromium-user-profile) Cookies & Sessions Setup  (https://github.com/ArchiveBox/ArchiveBox/wiki/Merging-Collections) Merging Collections  (https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting) Troubleshooting   More Info     ⭐️ (https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community) Web Archiving Community  (https://github.com/ArchiveBox/ArchiveBox#background--motivation) Background & Motivation  (https://github.com/ArchiveBox/ArchiveBox#comparison-to-other-projects) Comparison to Other Tools  (https://github.com/ArchiveBox/ArchiveBox/releases) Changelog & (https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap) Roadmap   (https://archivebox.io/)  (https://github.com/ArchiveBox/ArchiveBox)  (https://hcb.hackclub.com/donations/start/archivebox)  (https://github.com/sponsors/pirate)  (https://www.patreon.com/theSquashSH)  (https://zulip.archivebox.io/)      Clone this wiki locally (https://github.com/ArchiveBox/ArchiveBox.wiki.git)                     Footer (GitHub) (https://github.com/)    © 2024 GitHub, Inc.   Footer navigation (https://docs.github.com/site-policy/github-terms/github-terms-of-service) Terms  (https://docs.github.com/site-policy/privacy-policies/github-privacy-statement) Privacy  (https://github.com/security) Security  (https://www.githubstatus.com/) Status  (https://docs.github.com/) Docs  (https://support.github.com/?tags=dotcom-footer) Contact  Manage cookies    Do not share my personal information           You can’t perform that action at this time.