1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
|
File Generation Process
=======================
1 Tree Structure Data
1 Obtain data in otol/, as specified in it's README.
2 Run genOtolData.py, which creates data.db, and adds
'nodes' and 'edges' tables using data in otol/*, as well as
genOtolNamesToKeep.txt, if present.
2 Name Data for Search
1 Obtain data in eol/, as specified in it's README.
2 Run genEolNameData.py, which adds 'names' and 'eol\_ids' tables to data.db,
using data in eol/vernacularNames.csv and the 'nodes' table, and possibly
genEolNameDataPickedIds.txt.
3 Node Description Data
1 Obtain data in dbpedia/ and enwiki/, as specified in their README files.
2 Run genDbpData.py, which adds a 'descs' table to data.db, using
data in dbpedia/dbpData.db, the 'nodes' table, and possibly
genDescNamesToSkip.txt and dbpPickedLabels.txt.
3 Run genEnwikiDescData.py, which adds to the 'descs' table, using data in
enwiki/enwikiData.db, and the 'nodes' table. Also uses genDescNamesToSkip.txt and
genEnwikiDescTitlesToUse.txt for skipping/resolving some name-page associations.
3 Image Data
1 In eol/, run downloadImgs.py to download EOL images into eol/imgsForReview/.
It uses data in eol/imagesList.db, and the 'eol\_ids' table.
2 In eol/, run reviewImgs.py to filter images in eol/imgsForReview/ into EOL-id-unique
images in eol/imgsReviewed/ (uses 'names' and 'eol\_ids' to display extra info).
3 In enwiki/, run getEnwikiImgData.py, which generates a list of
tol-node images, and creates enwiki/enwikiImgs.db to store it.
Uses the 'descs' table to get tol-node wiki-ids.
4 In enwiki/, run downloadImgLicenseInfo.py, which downloads licensing
information for images listed in enwiki/enwikiImgs.db, and stores
it in that db.
5 In enwiki/, run downloadEnwikiImgs.py, which downloads 'permissively-licensed'
images in listed in enwiki/enwikiImgs.db, storing them in enwiki/imgs/.
6 Run reviewImgsToMerge.py, which displays images from eol/ and enwiki/,
enables choosing, for each tol-node, which image should be used, if any,
and outputs choice information into mergedImgList.txt.
7 Run genImgsForWeb.py, which creates cropped/resized images in img/, using
mergedImgList.txt, and adds 'images' and 'node_imgs' tables to data.db.
8 Run genLinkedImgs.py to add a 'linked_imgs' table to data.db,
which uses 'nodes', 'edges', 'eol\_ids', and 'node_imgs', to associate
nodes without images to child images.
5 Reduced Tree Structure Data
1 Run genReducedTreeData.py, which adds 'r_nodes' and 'r_edges' tables to
data.db, using reducedTol/names.txt, and the 'nodes' and 'names' tables.
6 Other
1 Can run genEnwikiNameData.py, which adds more entries to the 'names' table,
using data in enwiki/enwikiData.db, and the 'names' and 'descs' tables.
data.db Tables
==============
- nodes: name TEXT PRIMARY KEY, id TEXT UNIQUE, tips INT
- edges: node TEXT, child TEXT, p\_support INT, PRIMARY KEY (node, child)
- names: name TEXT, alt\_name TEXT, pref\_alt INT, PRIMARY KEY(name, alt\_name)
- eol\_ids: id INT PRIMARY KEY, name TEXT
- images: id INT, src TEXT, url TEXT, license TEXT, artist TEXT, credit TEXT, PRIMARY KEY (id, src)
- node\_imgs: id TEXT PRIMARY KEY, img\_id INT, src TEXT
- linked\_imgs: name TEXT PRIMARY KEY, eol\_id INT, eol\_id2 INT
- descs: name TEXT PRIMARY KEY, desc TEXT, redirected INT, wiki\_id INT, from\_dbp INT
- r\_nodes: name TEXT PRIMARY KEY, tips INT
- r\_edges: node TEXT, child TEXT, p\_support INT, PRIMARY KEY (node, child)
Other Files
===========
- dbpPickedLabels.txt <br>
Contains DBpedia labels, one per line. Used by genDbpData.py to help
resolve conflicts when associating tree-of-life node names with
DBpedia node labels.
- genOtolNamesToKeep.txt <br>
Contains names to avoid trimming off the tree data generated by
genOtolData.py. Usage is optional, but, without it, a large amount
of possibly-significant nodes are removed, using a short-sighted
heuristic. <br>
One way to generate this list is to generate the files as usual,
then get node names that have an associated image, description, or
presence in r_nodes. Then run the genOtolData.py and genEolNameData.py
scripts again (after deleting their created tables).
- genEnwikiDescNamesToSkip.txt <br>
Contains names for nodes that genEnwikiNameData.py should skip adding
a description for. Usage is optional, but without it, some nodes will
probably get descriptions that don't match (eg: the bee genus Osiris
might be described as an egyptian god). <br>
This file was generated by running genEnwikiNameData.py, then listing
the names that it added into a file, along with descriptions, and
manually removing those that seemed node-matching (got about 30k lines,
with about 1 in 30 descriptions non-matching). And, after creating
genEnwikiDescTitlesToUse.txt, names shared with that file were removed.
- genEnwikiDescTitlesToUse.txt <br>
Contains enwiki titles with the form 'name1 (category1)' for
genEnwikiNameData.py to use to resolve nodes matching name name1.
Usage is optional, but it adds some descriptions that would otherwise
be skipped. <br>
This file was generated by taking the content of genEnwikiNameData.py,
after the manual filtering step, then, for each name name,1 getting
page titles from dbpedia/dbpData.db that match 'name1 (category1)'.
This was followed by manually removing lines, keeping those that
seemed to match the corresponding node (used the app to help with this).
|
