From 5fe71ea7b9d9a5d2dc6e8e5ce5b9193629eed74d Mon Sep 17 00:00:00 2001 From: Terry Truong Date: Mon, 11 Jul 2022 01:54:08 +1000 Subject: Make backend dev server script serve the image files Previously, image files in backend/data/img were moved to, or symlinked from, public/. This needed to be changed before each build, otherwise vite would end up copying gigabytes of images. --- backend/data/enwiki/README.md | 52 ------------------------------------------- 1 file changed, 52 deletions(-) delete mode 100644 backend/data/enwiki/README.md (limited to 'backend/data/enwiki/README.md') diff --git a/backend/data/enwiki/README.md b/backend/data/enwiki/README.md deleted file mode 100644 index 90d16c7..0000000 --- a/backend/data/enwiki/README.md +++ /dev/null @@ -1,52 +0,0 @@ -This directory holds files obtained from/using [English Wikipedia](https://en.wikipedia.org/wiki/Main_Page). - -# Downloaded Files -- enwiki-20220501-pages-articles-multistream.xml.bz2
- Obtained via (site suggests downloading from a mirror). - Contains text content and metadata for pages in enwiki. - Some file content and format information was available from - . -- enwiki-20220501-pages-articles-multistream-index.txt.bz2
- Obtained like above. Holds lines of the form offset1:pageId1:title1, - providing, for each page, an offset into the dump file of a chunk of - 100 pages that includes it. - -# Generated Dump-Index Files -- genDumpIndexDb.py
- Creates an sqlite-database version of the enwiki-dump index file. -- dumpIndex.db
- Generated by genDumpIndexDb.py.
- Tables:
- - `offsets`: `title TEXT PRIMARY KEY, id INT UNIQUE, offset INT, next_offset INT` - -# Description Database Files -- genDescData.py
- Reads through pages in the dump file, and adds short-description info to a database. -- descData.db
- Generated by genDescData.py.
- Tables:
- - `pages`: `id INT PRIMARY KEY, title TEXT UNIQUE` - - `redirects`: `id INT PRIMARY KEY, target TEXT` - - `descs`: `id INT PRIMARY KEY, desc TEXT` - -# Image Database Files -- genImgData.py
- Used to find infobox image names for page IDs, storing them into a database. -- downloadImgLicenseInfo.py
- Used to download licensing metadata for image names, via wikipedia's online API, storing them into a database. -- imgData.db
- Used to hold metadata about infobox images for a set of pageIDs. - Generated using getEnwikiImgData.py and downloadImgLicenseInfo.py.
- Tables:
- - `page_imgs`: `page_id INT PRIMAY KEY, img_name TEXT`
- `img_name` may be null, which means 'none found', and is used to avoid re-processing page-ids. - - `imgs`: `name TEXT PRIMARY KEY, license TEXT, artist TEXT, credit TEXT, restrictions TEXT, url TEXT`
- Might lack some matches for `img_name` in `page_imgs`, due to licensing info unavailability. -- downloadImgs.py
- Used to download image files into imgs/. - -# Other Files -- lookupPage.py
- Running `lookupPage.py title1` looks in the dump for a page with a given title, - and prints the contents to stdout. Uses dumpIndex.db. - -- cgit v1.2.3