aboutsummaryrefslogtreecommitdiff
path: root/backend/data
AgeCommit message (Collapse)Author
2022-06-03Add script for extracting enwiki infobox-image namesTerry Truong
2022-06-03Adjust enwiki dump-index-db and lookup script to include wiki-idsTerry Truong
2022-06-03Rename some scripts to be eol-specific indicativeTerry Truong
2022-06-01Add script/instructions for filtering out mismatching node descsTerry Truong
2022-05-28Convert from python "...".format() to f"..."Terry Truong
2022-05-28Fix tree-trimming to not discard certain nodesTerry Truong
Was trimming off nodes without an image/desc/reduced_tree_presence, including those with a descdendant that has them. Was using a linked-image association to prevent this, but this wasn't reliable.
2022-05-26Fix missed space-to-underscore conversion in lookupPage.pyTerry Truong
2022-05-26Add some enwiki redirect data as alt-namesTerry Truong
2022-05-26Trim otol tree to avoid certain slowdownsTerry Truong
Some nodes had multiple ancestors with over 10k children, and jump-searching to them could take almost a minute for vue to load.
2022-05-24For compound-nodes without sub-images, don't use a child imageTerry Truong
2022-05-24Don't generate compound-image if both sub-images non-existentTerry Truong
2022-05-23Display 'compound images'Terry Truong
Adjust genLinkedImgs.py to associate multiple images to compound-nodes. Adjust server to send multiple image filenames for such nodes. Adjust Tile and TileInfoModal to display those compound-images.
2022-05-22Up-propgate images for usage in parents without imagesTerry Truong
Also adjust vite config to avoid apparent redundant processing when images are being downloaded.
2022-05-19Unescape EOL alt-name dataTerry Truong
2022-05-19Fix not scrolling to focused nodesTerry Truong
2022-05-18Add wikipedia-link to tile-info displayTerry Truong
Add 'wiki_id' and 'from_dbp' columns to 'descs' table, adjust dbpedia data to include wikipedia IDs, adjust data generations scripts, make server send extra data, and make TileInfo display it.
2022-05-18For dbpedia data, prioritise manual conflict resolutionsTerry Truong
2022-05-17Make reduced tree generation check non-alt-namesTerry Truong
2022-05-17Remove certain same-name alt-namesTerry Truong
2022-05-17Make search suggestions include non-alt-namesTerry Truong
2022-05-17Fix small genDbpData.py typoTerry Truong
2022-05-17Improve enwiki description extractionTerry Truong
Adjust enwiki code to handle single dump file, and add scripts for 'convenient' page-content lookup.
2022-05-15Converted nodes+r_nodes tables into nodes+edges+r_nodes+r_edgesTerry Truong
Conversion avoids encoding node children as JSON strings, and allows for easier querying of edge data. Adjusted server to use new format. Also added some table indexes for common operations.
2022-05-14Add desc-resolution special-case for root nodeTerry Truong
2022-05-14Supplement DBpedia descs with enwiki'sTerry Truong
2022-05-14Use DBpedia data for node descriptionsTerry Truong
Add backend/data/dbpedia/ directory containing scripts and README for obtaining DBpedia data, storing it into a db, converting/adding description data to data.db, and for resolving tol-node DBpedia-node association conflicts (via DBpedia relations, manual listing, etc). Resulted in less (about 3/4 as many) descriptions as with using enwiki, but with notably less mis-associations (eg: node Thor is described as a shrimp instead of a god).
2022-05-12Adjust reviewImgs.py to remove EXIF orientation dataTerry Truong
2022-05-12Commit reduced-tree name-list dataTerry Truong
2022-05-12Add reduced-tree data generation+serving+querying+settingTerry Truong
Add genReducedTreeData.py, which generates a reduced_nodes table. Adjust server to serve that data for queries with a tree=reduced query param. Adjust client to query for that data depending on a useReducedTree variable. Add a SettingsPane setting to change that useReducedTree variable.
2022-05-11Do minor code cleanupTerry Truong
2022-05-11Use prefix-search with ranking-by-lengthTerry Truong
2022-05-10Add quotes around displayed alt-namesTerry Truong
2022-05-07Add timing info display to reviewImgs.pyTerry Truong
2022-05-07Update README line breaksTerry Truong
2022-05-06Fix typo in data entry codeTerry Truong
2022-05-05Make '... [N]' duplicate node name assignment prefer nodes with less tipsTerry Truong
Adjust genOtolData.py to use int ids instead of names, reducing memory usage. Small fix for server.py commented-out substring-search code.
2022-05-05Add db table info to data/README.mdTerry Truong
2022-05-04Fix forgetting to commit imagesList.db generator scriptTerry Truong
2022-05-04Add scripts for obtaining/sending/displaying wikipedia descriptionsTerry Truong
Add backend/data/enwiki/ directory containing scripts and instructive READMEs. Adjust some other scripts to generate 'eol_ids' sqlite table separate from 'names'. Make server respond to /data/desc requests, and have client TileInfo component display response data. Also adjust .gitignore entries to be root-relative.
2022-05-01Fix non-inclusion of canonical-names without english alt-namesTerry Truong
Also fix lack of placeholder-image display in tile-info modal.
2022-05-01Make name data English-specificTerry Truong
2022-05-01Add fuzzy-search via sqlite extension spellfix1Terry Truong
Also add delay between client search-suggestion requests when search input undergoes multiple quick changes
2022-05-01Make node data in db not collapsed-in-jsonTerry Truong
2022-04-30Add preferred-alt-name data to db, and use when reviewingTerry Truong
2022-04-30Add scripts for downloading/reviewing/cropping_and_resizing imagesTerry Truong
Also adjust client code to handle new format, and add backend/data/README.md explaining image production process.
2022-04-27Make generated EOL-names-data have unique eol_id for each tolnode nameTerry Truong
2022-04-27Use EOL vernacular-names data for searchingTerry Truong
Add data/eolNamesToSqlite.py to read EOL vernacular-names data and add to sqlite db. Adjust server.py to handle search requests, and adjust SearchModal to make them.
2022-04-26Change data URL path formatTerry Truong
2022-04-26Add small sqrLayout optimisationTerry Truong
2022-04-26Have tol data in sqlite db, and add server script that accesses itTerry Truong
Adapt otol-data-converting script to generate otol.db, add server.py script that provides access to that db, and adapt the app to query the server for tol data when needed.