From 436dd015471cbdea443cfd98536e55e683833c48 Mon Sep 17 00:00:00 2001 From: Terry Truong Date: Sun, 22 Jan 2023 11:32:28 +1100 Subject: Add deployment docs and script Add DEPLOY.md and prebuild.sh Update READMEs Change project name --- DEPLOY.md | 39 +++++ README.md | 49 ++++-- backend/README.md | 2 +- backend/chrona.py | 417 ++++++++++++++++++++++++++++++++++++++++++++++++++ backend/histplorer.py | 417 -------------------------------------------------- backend/server.py | 2 +- index.html | 2 +- package.json | 2 +- prebuild.sh | 7 + src/README.md | 19 ++- src/lib.ts | 2 +- 11 files changed, 521 insertions(+), 437 deletions(-) create mode 100644 DEPLOY.md create mode 100755 backend/chrona.py delete mode 100755 backend/histplorer.py create mode 100755 prebuild.sh diff --git a/DEPLOY.md b/DEPLOY.md new file mode 100644 index 0000000..98c4077 --- /dev/null +++ b/DEPLOY.md @@ -0,0 +1,39 @@ +# Instructions for Deployment on an Apache server (version 2.4) on Ubuntu (22.04.1 LTS) + +1. Set up the server environment + - If Python3 and jsonpickle aren't installed, this can be done using + `apt-get update; apt-get install python3 python3-jsonpickle`. + - Install `mod_wsgi` by running `apt-get install libapache2-mod-wsgi-py3`. This is an Apache module for WSGI. + It's for running `backend/chrona.py` to serve tree-of-life data, and is used instead of CGI to avoid + starting a new process for each request. +1. Change some constants (automated by `prebuild.sh`) + - In `src/vite.config.js`: Set `base` to the URL path where Chrona will be accessible (eg: `'/chrona'`) + - In `src/lib.ts`: + - Set `SERVER_DATA_URL` to the URL where `backend/chrona.py` will be served + (eg: `'https://terryt.dev/chrona/data'`) + - Set `SERVER_IMG_PATH` to the URL path where images will be served (eg: `'/img/chrona'`). + If you place it within the `base` directory, you'll need to remember to move it when deploying + a newer production build. + - In `backend/chrona.py`: Set `DB_FILE` to where the database will be placed (eg: `'/usr/local/www/db/chrona.db'`) +1. Generate the client-side production build
+ Run `npm run build`. This generates a directory `dist/`. +1. Copy files to the server (using ssh, sftp, or otherwise) + 1. Copy `dist/` into Apache's document root, into the directory where Chrona will be served. + The created directory should match up with the `base` value above (eg: `/var/www/terryt.dev/chrona/`). + 1. Copy over `backend/chrona.py`. The location should be accessible by Apache (eg: `/usr/local/www/wsgi-scripts/`). + Remember to set ownership and permissions as needed. + 1. Copy over `backend/hist_data/data.db`. The result should be denoted by the `DB_FILE` value above. + 1. Copy over the images in `backend/hist_data/img/`. There are a lot of them, so compressing them + before transfer is advisable (eg: `tar czf imgs.tar.gz backend/hist_data/img/`). The location should + match up with the `SERVER_IMG_PATH` value above (eg: `/var/www/terryt.dev/img/chrona/`). + 1. Edit the site's config file to serve chrona.py. The file path will likely be something like + `/etc/apache2/sites-available/terryt.dev-le-ssl.conf`, and the edit should add lines like the following, + likely within a `` section: + + WSGIScriptAlias /chrona/data /usr/local/www/wsgi-scripts/chrona.py + + Require all granted + + + The first `WSGIScriptAlias` parameter should match the URL path in `SERVER_URL`, and the second should + be the location of chrona.py. The `` lines enable access for that location. diff --git a/README.md b/README.md index 3d98947..2929759 100644 --- a/README.md +++ b/README.md @@ -1,12 +1,18 @@ -# Histplorer +# Chrona An interactive historical timeline. -[Available online](https://terryt.dev/tilo/). +Available online. ## Project Overview -Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod -tempor incididunt ut labore et dolore magna aliqua. +The UI is largely coded in Typescript, using the [Vue](https://vuejs.org) +framework, with [Vite](https://vitejs.dev) as the build tool. Much of +the styling is done using [Tailwind](https://tailwindcss.com). Packages +are managed using [npm](https://www.npmjs.com) and [Node.js](https://nodejs.org). + +On the server side, tree data is served and generated using Python, with +packages managed using [Pip](https://pypi.org/project/pip). Tree data is +stored using [Sqlite](https://www.sqlite.org). ## Files @@ -16,6 +22,7 @@ tempor incididunt ut labore et dolore magna aliqua. - **LICENCE.txt**: This project's license (MIT) ### Client & Server - **src**: Contains most of the client-side code +- **tests**: Contains client-side unit tests - **index.html**: Holds code for the main page, into which code from 'src' will be included - **public**: Contains files to be copied unchanged in the client's production build - **backend**: Contains code for the server, and for generating history data @@ -25,30 +32,46 @@ tempor incididunt ut labore et dolore magna aliqua. - **postcss.config.js**: For configuring Tailwind - **tsconfig.json**: For configuring Typescript - **tsconfig.node.json**: For configuring Typescript +- **.eslintrc.js**: For configuring ESLint ### Other - **.gitignore**: Lists files to be ignored if using Git +- **DEPLOY.md**: Instructions for deployment on an Apache server on Ubuntu. +- **prebuild.sh**: Bash script for automating some steps of deployment. ## Setup Instructions +Note: Running your own version of the client and server should be straightforward, +but generating the database takes a long time. +More details are in `backend/hist_data/README.md`. + ### Client Side -Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod -tempor incididunt ut labore et dolore magna aliqua. +1. If you don't have npm or Node.js installed, you can download a Node installer from + , which includes npm. This project was coded using version 16. +1. In this directory, run the command `npm install`, which install packages listed in + package.json, creating a `node_modules` directory to hold them. ### Server Side -Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod -tempor incididunt ut labore et dolore magna aliqua. +1. If you don't have Python 3 installed, see . + The package manager Pip is included. +1. The database used by the server is generated using scripts in `backend/hist_data/`. + See it's README for instructions. You'll likely need to install a few + packages using Pip. +1. To run the data server via `backend/server.py`, you'll need to install jsonpickle. + This can be done using `python -m pip install jsonpickle`. -### Running Histplorer + If you want to keep the installed package separate from your system's packages, + it's common practice to use [venv](https://docs.python.org/3/tutorial/venv.html). + +### Running Chrona 1. In `backend/`, run `./server.py`, which starts a basic HTTP server that provides history data on port 8000. 1. Running `npm run dev` starts the dev server. 1. Open a web browser, and navigate to . ## Deploying the Website - -Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod -tempor incididunt ut labore et dolore magna aliqua. +This is significantly dependent on the server platform. `DEPLOY.md` contains +instructions for deployment on an Apache server on an Ubuntu system. ## Licence -Histplorer is licensed under the MIT Licence. +Chrona is licensed under the MIT Licence. diff --git a/backend/README.md b/backend/README.md index cffbb81..6c7db61 100644 --- a/backend/README.md +++ b/backend/README.md @@ -1,6 +1,6 @@ # Files - `hist_data/`: Holds scripts for generating the history database and images -- `histplorer.py`: WSGI script that serves data from the history database +- `chrona.py`: WSGI script that serves data from the history database - `server.py`: Basic dev server that serves the WSGI script and image files - `tests/`: Holds unit testing scripts
Running all tests: `python -m unittest discover -s tests`
diff --git a/backend/chrona.py b/backend/chrona.py new file mode 100755 index 0000000..cd9a0be --- /dev/null +++ b/backend/chrona.py @@ -0,0 +1,417 @@ +""" +WSGI script that serves historical data. + +Expected HTTP query parameters: +- type: + If 'events', reply with information on events within a date range, for a given scale. + If 'info', reply with information about a given event. + If 'sugg', reply with search suggestions for an event search string. +- range: With type=events, specifies a historical-date range. + If absent, the default is 'all of time'. + Examples: + range=1000.1910-10-09 means '1000 AD up to and excluding 09/10/1910' + range=-13000. means '13000 BC onwards' +- scale: With type=events, specifies a date scale (see SCALES in hist_data/cal.py). +- incl: With type=events, specifies an event to include, as an event ID. +- event: With type=info, specifies the title of an event to get info for. +- input: With type=sugg, specifies a search string to suggest for. +- limit: With type=events or type=sugg, specifies the max number of results. +- ctgs: With type=events|info|sugg, specifies event categories to restrict results to. + Interpreted as a period-separated list of category names (eg: person.place). + An empty string is ignored. +- imgonly: With type=events|info|sugg, if present, restricts results to events with images. +""" + +from typing import Iterable, cast +import sys +import re +import urllib.parse +import sqlite3 +import gzip +import jsonpickle + +from hist_data.cal import HistDate, dbDateToHistDate, dateToUnit + +DB_FILE = 'hist_data/data.db' +MAX_REQ_EVENTS = 500 +MAX_REQ_UNIT_COUNTS = MAX_REQ_EVENTS +DEFAULT_REQ_EVENTS = 20 +MAX_REQ_SUGGS = 50 +DEFAULT_REQ_SUGGS = 5 + +# ========== Classes for values sent as responses ========== + +class HistEvent: + """ Represents an historical event """ + def __init__( + self, + id: int, + title: str, + start: HistDate, + startUpper: HistDate | None, + end: HistDate | None, + endUpper: HistDate | None, + ctg: str, + imgId: int | None, + pop: int): + self.id = id + self.title = title + self.start = start + self.startUpper = startUpper + self.end = end + self.endUpper = endUpper + self.ctg = ctg + self.imgId = imgId + self.pop = pop + + def __eq__(self, other): # Used in unit testing + return isinstance(other, HistEvent) and \ + (self.id, self.title, self.start, self.startUpper, self.end, self.endUpper, \ + self.ctg, self.pop, self.imgId) == \ + (other.id, other.title, other.start, other.startUpper, other.end, other.endUpper, \ + other.ctg, other.pop, other.imgId) + + def __repr__(self): # Used in unit testing + return str(self.__dict__) + +class EventResponse: + """ Used when responding to type=events requests """ + def __init__(self, events: list[HistEvent], unitCounts: dict[int, int] | None): + self.events = events + self.unitCounts = unitCounts # None indicates exceeding MAX_REQ_UNIT_COUNTS + + def __eq__(self, other): # Used in unit testing + return isinstance(other, EventResponse) and \ + (self.events, self.unitCounts) == (other.events, other.unitCounts) + + def __repr__(self): # Used in unit testing + return str(self.__dict__) + +class ImgInfo: + """ Represents an event's associated image """ + def __init__(self, url: str, license: str, artist: str, credit: str): + self.url = url + self.license = license + self.artist = artist + self.credit = credit + + def __eq__(self, other): # Used in unit testing + return isinstance(other, ImgInfo) and \ + (self.url, self.license, self.artist, self.credit) == \ + (other.url, other.license, other.artist, other.credit) + + def __repr__(self): # Used in unit testing + return str(self.__dict__) + +class EventInfo: + """ Used when responding to type=info requests """ + def __init__(self, event: HistEvent, desc: str | None, wikiId: int, imgInfo: ImgInfo | None): + self.event = event + self.desc = desc + self.wikiId = wikiId + self.imgInfo = imgInfo + + def __eq__(self, other): # Used in unit testing + return isinstance(other, EventInfo) and \ + (self.event, self.desc, self.wikiId, self.imgInfo) == (other.event, other.desc, other.wikiId, other.imgInfo) + + def __repr__(self): # Used in unit testing + return str(self.__dict__) + +class SuggResponse: + """ Used when responding to type=sugg requests """ + def __init__(self, suggs: list[str], hasMore: bool): + self.suggs = suggs + self.hasMore = hasMore + + def __eq__(self, other): # Used in unit testing + return isinstance(other, SuggResponse) and \ + (self.suggs, self.hasMore) == (other.suggs, other.hasMore) + + def __repr__(self): # Used in unit testing + return str(self.__dict__) + +# ========== Entry point ========== + +def application(environ: dict[str, str], start_response) -> Iterable[bytes]: + """ Entry point for the WSGI script """ + # Get response object + val = handleReq(DB_FILE, environ) + + # Construct response + data = jsonpickle.encode(val, unpicklable=False).encode() + headers = [('Content-type', 'application/json')] + if 'HTTP_ACCEPT_ENCODING' in environ and 'gzip' in environ['HTTP_ACCEPT_ENCODING']: + if len(data) > 100: + data = gzip.compress(data, compresslevel=5) + headers.append(('Content-encoding', 'gzip')) + headers.append(('Content-Length', str(len(data)))) + start_response('200 OK', headers) + + return [data] + +def handleReq(dbFile: str, environ: dict[str, str]) -> None | EventResponse | EventInfo | SuggResponse: + """ Queries the database, and constructs a response object """ + # Open db + dbCon = sqlite3.connect(dbFile) + dbCur = dbCon.cursor() + + # Get query params + queryStr = environ['QUERY_STRING'] if 'QUERY_STRING' in environ else '' + queryDict = urllib.parse.parse_qs(queryStr) + params = {k: v[0] for k, v in queryDict.items()} + + # Get data of requested type + reqType = queryDict['type'][0] if 'type' in queryDict else None + if reqType == 'events': + return handleEventsReq(params, dbCur) + elif reqType == 'info': + return handleInfoReq(params, dbCur) + elif reqType == 'sugg': + return handleSuggReq(params, dbCur) + return None + +# ========== For handling type=events ========== + +def handleEventsReq(params: dict[str, str], dbCur: sqlite3.Cursor) -> EventResponse | None: + """ Generates a response for a type=events request """ + # Get dates + dateRange = params['range'] if 'range' in params else '.' + if '.' not in dateRange: + print(f'INFO: Invalid date-range value {dateRange}', file=sys.stderr) + return None + try: + start, end = [reqParamToHistDate(s) for s in dateRange.split('.', maxsplit=1)] + except ValueError: + print(f'INFO: Invalid date-range value {dateRange}', file=sys.stderr) + return None + + # Get scale + if 'scale' not in params: + print('INFO: No scale provided', file=sys.stderr) + return None + try: + scale = int(params['scale']) + except ValueError: + print('INFO: Invalid scale value', file=sys.stderr) + return None + + # Get incl value + try: + incl = int(params['incl']) if 'incl' in params else None + except ValueError: + print('INFO: Invalid incl value', file=sys.stderr) + return None + + # Get result set limit + try: + resultLimit = int(params['limit']) if 'limit' in params else DEFAULT_REQ_EVENTS + except ValueError: + print(f'INFO: Invalid results limit {resultLimit}', file=sys.stderr) + return None + if resultLimit <= 0 or resultLimit > MAX_REQ_EVENTS: + print(f'INFO: Invalid results limit {resultLimit}', file=sys.stderr) + return None + + ctgs = params['ctgs'].split('.') if 'ctgs' in params else None + imgonly = 'imgonly' in params + + events = lookupEvents(start, end, scale, incl, resultLimit, ctgs, imgonly, dbCur) + unitCounts = lookupUnitCounts(start, end, scale, imgonly, dbCur) + + return EventResponse(events, unitCounts) + +def reqParamToHistDate(s: str): + """ Produces a HistDate from strings like '2010-10-3', '-8000', and '' (throws ValueError if invalid) """ + if not s: + return None + m = re.match(r'(-?\d+)(?:-(\d+)-(\d+))?', s) + if m is None: + raise ValueError('Invalid HistDate string') + if m.lastindex == 1: + return HistDate(None, int(m.group(1))) + else: + return HistDate(True, int(m.group(1)), int(m.group(2)), int(m.group(3))) + +def lookupEvents( + start: HistDate | None, end: HistDate | None, scale: int, incl: int | None, resultLimit: int, + ctgs: list[str] | None, imgonly: bool, dbCur: sqlite3.Cursor) -> list[HistEvent]: + """ Looks for events within a date range, in given scale, + restricted by event category, an optional particular inclusion, and a result limit """ + dispTable = 'event_disp' if not imgonly else 'img_disp' + query = \ + 'SELECT events.id, title, start, start_upper, end, end_upper, fmt, ctg, images.id, pop.pop FROM events' \ + f' INNER JOIN {dispTable} ON events.id = {dispTable}.id' \ + ' INNER JOIN pop ON events.id = pop.id' \ + ' LEFT JOIN event_imgs ON events.id = event_imgs.id' \ + ' LEFT JOIN images ON event_imgs.img_id = images.id' + constraints = [f'{dispTable}.scale = ?'] + params: list[str | int] = [scale] + + # Constrain by start/end + startUnit = dateToUnit(start, scale) if start is not None else None + endUnit = dateToUnit(end, scale) if end is not None else None + if startUnit is not None and startUnit == endUnit: + constraints.append(f'{dispTable}.unit = ?') + params.append(startUnit) + else: + if startUnit is not None: + constraints.append(f'{dispTable}.unit >= ?') + params.append(startUnit) + if endUnit is not None: + constraints.append(f'{dispTable}.unit < ?') + params.append(endUnit) + + # Constrain by event category + if ctgs is not None: + constraints.append('ctg IN (' + ','.join('?' * len(ctgs)) + ')') + params.extend(ctgs) + + # Add constraints to query + query2 = query + if constraints: + query2 += ' WHERE ' + ' AND '.join(constraints) + query2 += ' ORDER BY pop.pop DESC' + query2 += f' LIMIT {resultLimit}' + + # Run query + results: list[HistEvent] = [] + for row in dbCur.execute(query2, params): + results.append(eventEntryToResults(row)) + if incl is not None and incl == row[0]: + incl = None + + # Get any additional inclusion + if incl is not None: + row = dbCur.execute(query + ' WHERE events.id = ?', (incl,)).fetchone() + if row is not None: + if len(results) == resultLimit: + results.pop() + results.append(eventEntryToResults(row)) + + return results + +def eventEntryToResults( + row: tuple[int, str, int, int | None, int | None, int | None, int, str, int | None, int]) -> HistEvent: + eventId, title, start, startUpper, end, endUpper, fmt, ctg, imageId, pop = row + """ Helper for converting an 'events' db entry into an HistEvent object """ + # Convert dates + dateVals: list[int | None] = [start, startUpper, end, endUpper] + newDates: list[HistDate | None] = [None for n in dateVals] + for i, n in enumerate(dateVals): + if n is not None: + newDates[i] = dbDateToHistDate(n, fmt, i < 2) + + return HistEvent( + eventId, title, cast(HistDate, newDates[0]), newDates[1], newDates[2], newDates[3], ctg, imageId, pop) + +def lookupUnitCounts( + start: HistDate | None, end: HistDate | None, scale: int, + imgonly: bool, dbCur: sqlite3.Cursor) -> dict[int, int] | None: + """ Return list of units with counts given scale and a date range """ + # Build query + distTable = 'dist' if not imgonly else 'img_dist' + query = f'SELECT unit, count FROM {distTable} WHERE scale = ?' + params = [scale] + if start: + query += ' AND unit >= ?' + params.append(dateToUnit(start, scale)) + if end: + query += ' AND unit < ?' + params.append(dateToUnit(end, scale)) + query += ' ORDER BY unit ASC LIMIT ' + str(MAX_REQ_UNIT_COUNTS + 1) + + # Get results + unitCounts: dict[int, int] = {} + for unit, count in dbCur.execute(query, params): + unitCounts[unit] = count + return unitCounts if len(unitCounts) <= MAX_REQ_UNIT_COUNTS else None + +# ========== For handling type=info ========== + +def handleInfoReq(params: dict[str, str], dbCur: sqlite3.Cursor): + """ Generates a response for a type=info request """ + if 'event' not in params: + print('INFO: No \'event\' parameter for type=info request', file=sys.stderr) + return None + ctgs = params['ctgs'].split('.') if 'ctgs' in params else None + imgonly = 'imgonly' in params + return lookupEventInfo(params['event'], ctgs, imgonly, dbCur) + +def lookupEventInfo(eventTitle: str, ctgs: list[str] | None, imgonly: bool, dbCur: sqlite3.Cursor) -> EventInfo | None: + """ Look up an event with given title, and return a descriptive EventInfo """ + imgJoin = 'INNER JOIN' if imgonly else 'LEFT JOIN' + query = \ + 'SELECT events.id, title, start, start_upper, end, end_upper, fmt, ctg, images.id, pop.pop, ' \ + ' descs.desc, descs.wiki_id, ' \ + ' images.url, images.license, images.artist, images.credit FROM events' \ + ' INNER JOIN pop ON events.id = pop.id' \ + f' {imgJoin} event_imgs ON events.id = event_imgs.id' \ + f' {imgJoin} images ON event_imgs.img_id = images.id' \ + ' LEFT JOIN descs ON events.id = descs.id' \ + ' WHERE events.title = ? COLLATE NOCASE' + row = dbCur.execute(query, (eventTitle,)).fetchone() + if row is not None: + event = eventEntryToResults(row[:10]) + desc, wikiId, url, license, artist, credit = row[10:] + if ctgs is not None and event.ctg not in ctgs: + return None + return EventInfo(event, desc, wikiId, None if url is None else ImgInfo(url, license, artist, credit)) + else: + return None + +# ========== For handling type=sugg ========== + +def handleSuggReq(params: dict[str, str], dbCur: sqlite3.Cursor): + """ Generates a response for a type=sugg request """ + # Get search string + if 'input' not in params: + print('INFO: No \'input\' parameter for type=sugg request', file=sys.stderr) + return None + searchStr = params['input'] + if not searchStr: + print('INFO: Empty \'input\' parameter for type=sugg request', file=sys.stderr) + return None + + # Get result limit + try: + resultLimit = int(params['limit']) if 'limit' in params else DEFAULT_REQ_SUGGS + except ValueError: + print(f'INFO: Invalid suggestion limit {resultLimit}', file=sys.stderr) + return None + if resultLimit <= 0 or resultLimit > MAX_REQ_SUGGS: + print(f'INFO: Invalid suggestion limit {resultLimit}', file=sys.stderr) + return None + + ctgs = params['ctgs'].split('.') if 'ctgs' in params else None + imgonly = 'imgonly' in params + return lookupSuggs(searchStr, resultLimit, ctgs, imgonly, dbCur) + +def lookupSuggs( + searchStr: str, resultLimit: int, ctgs: list[str] | None, imgonly: bool, dbCur: sqlite3.Cursor) -> SuggResponse: + """ For a search string, returns a SuggResponse describing search suggestions """ + tempLimit = resultLimit + 1 # For determining if 'more suggestions exist' + query = 'SELECT title FROM events LEFT JOIN pop ON events.id = pop.id' \ + + (' INNER JOIN event_imgs ON events.id = event_imgs.id' if imgonly else '') \ + + ' WHERE title LIKE ?' + if ctgs is not None: + query += ' AND ctg IN (' + ','.join('?' * len(ctgs)) + ')' + query += f' ORDER BY pop.pop DESC LIMIT {tempLimit}' + suggs: list[str] = [] + + # Prefix search + params = [searchStr + '%'] + (ctgs if ctgs is not None else []) + for (title,) in dbCur.execute(query, params): + suggs.append(title) + + # If insufficient results, try substring search + if len(suggs) < tempLimit: + existing = set(suggs) + params = ['%' + searchStr + '%'] + (ctgs if ctgs is not None else []) + for (title,) in dbCur.execute(query, params): + if title not in existing: + suggs.append(title) + if len(suggs) == tempLimit: + break + + return SuggResponse(suggs[:resultLimit], len(suggs) > resultLimit) diff --git a/backend/histplorer.py b/backend/histplorer.py deleted file mode 100755 index cd9a0be..0000000 --- a/backend/histplorer.py +++ /dev/null @@ -1,417 +0,0 @@ -""" -WSGI script that serves historical data. - -Expected HTTP query parameters: -- type: - If 'events', reply with information on events within a date range, for a given scale. - If 'info', reply with information about a given event. - If 'sugg', reply with search suggestions for an event search string. -- range: With type=events, specifies a historical-date range. - If absent, the default is 'all of time'. - Examples: - range=1000.1910-10-09 means '1000 AD up to and excluding 09/10/1910' - range=-13000. means '13000 BC onwards' -- scale: With type=events, specifies a date scale (see SCALES in hist_data/cal.py). -- incl: With type=events, specifies an event to include, as an event ID. -- event: With type=info, specifies the title of an event to get info for. -- input: With type=sugg, specifies a search string to suggest for. -- limit: With type=events or type=sugg, specifies the max number of results. -- ctgs: With type=events|info|sugg, specifies event categories to restrict results to. - Interpreted as a period-separated list of category names (eg: person.place). - An empty string is ignored. -- imgonly: With type=events|info|sugg, if present, restricts results to events with images. -""" - -from typing import Iterable, cast -import sys -import re -import urllib.parse -import sqlite3 -import gzip -import jsonpickle - -from hist_data.cal import HistDate, dbDateToHistDate, dateToUnit - -DB_FILE = 'hist_data/data.db' -MAX_REQ_EVENTS = 500 -MAX_REQ_UNIT_COUNTS = MAX_REQ_EVENTS -DEFAULT_REQ_EVENTS = 20 -MAX_REQ_SUGGS = 50 -DEFAULT_REQ_SUGGS = 5 - -# ========== Classes for values sent as responses ========== - -class HistEvent: - """ Represents an historical event """ - def __init__( - self, - id: int, - title: str, - start: HistDate, - startUpper: HistDate | None, - end: HistDate | None, - endUpper: HistDate | None, - ctg: str, - imgId: int | None, - pop: int): - self.id = id - self.title = title - self.start = start - self.startUpper = startUpper - self.end = end - self.endUpper = endUpper - self.ctg = ctg - self.imgId = imgId - self.pop = pop - - def __eq__(self, other): # Used in unit testing - return isinstance(other, HistEvent) and \ - (self.id, self.title, self.start, self.startUpper, self.end, self.endUpper, \ - self.ctg, self.pop, self.imgId) == \ - (other.id, other.title, other.start, other.startUpper, other.end, other.endUpper, \ - other.ctg, other.pop, other.imgId) - - def __repr__(self): # Used in unit testing - return str(self.__dict__) - -class EventResponse: - """ Used when responding to type=events requests """ - def __init__(self, events: list[HistEvent], unitCounts: dict[int, int] | None): - self.events = events - self.unitCounts = unitCounts # None indicates exceeding MAX_REQ_UNIT_COUNTS - - def __eq__(self, other): # Used in unit testing - return isinstance(other, EventResponse) and \ - (self.events, self.unitCounts) == (other.events, other.unitCounts) - - def __repr__(self): # Used in unit testing - return str(self.__dict__) - -class ImgInfo: - """ Represents an event's associated image """ - def __init__(self, url: str, license: str, artist: str, credit: str): - self.url = url - self.license = license - self.artist = artist - self.credit = credit - - def __eq__(self, other): # Used in unit testing - return isinstance(other, ImgInfo) and \ - (self.url, self.license, self.artist, self.credit) == \ - (other.url, other.license, other.artist, other.credit) - - def __repr__(self): # Used in unit testing - return str(self.__dict__) - -class EventInfo: - """ Used when responding to type=info requests """ - def __init__(self, event: HistEvent, desc: str | None, wikiId: int, imgInfo: ImgInfo | None): - self.event = event - self.desc = desc - self.wikiId = wikiId - self.imgInfo = imgInfo - - def __eq__(self, other): # Used in unit testing - return isinstance(other, EventInfo) and \ - (self.event, self.desc, self.wikiId, self.imgInfo) == (other.event, other.desc, other.wikiId, other.imgInfo) - - def __repr__(self): # Used in unit testing - return str(self.__dict__) - -class SuggResponse: - """ Used when responding to type=sugg requests """ - def __init__(self, suggs: list[str], hasMore: bool): - self.suggs = suggs - self.hasMore = hasMore - - def __eq__(self, other): # Used in unit testing - return isinstance(other, SuggResponse) and \ - (self.suggs, self.hasMore) == (other.suggs, other.hasMore) - - def __repr__(self): # Used in unit testing - return str(self.__dict__) - -# ========== Entry point ========== - -def application(environ: dict[str, str], start_response) -> Iterable[bytes]: - """ Entry point for the WSGI script """ - # Get response object - val = handleReq(DB_FILE, environ) - - # Construct response - data = jsonpickle.encode(val, unpicklable=False).encode() - headers = [('Content-type', 'application/json')] - if 'HTTP_ACCEPT_ENCODING' in environ and 'gzip' in environ['HTTP_ACCEPT_ENCODING']: - if len(data) > 100: - data = gzip.compress(data, compresslevel=5) - headers.append(('Content-encoding', 'gzip')) - headers.append(('Content-Length', str(len(data)))) - start_response('200 OK', headers) - - return [data] - -def handleReq(dbFile: str, environ: dict[str, str]) -> None | EventResponse | EventInfo | SuggResponse: - """ Queries the database, and constructs a response object """ - # Open db - dbCon = sqlite3.connect(dbFile) - dbCur = dbCon.cursor() - - # Get query params - queryStr = environ['QUERY_STRING'] if 'QUERY_STRING' in environ else '' - queryDict = urllib.parse.parse_qs(queryStr) - params = {k: v[0] for k, v in queryDict.items()} - - # Get data of requested type - reqType = queryDict['type'][0] if 'type' in queryDict else None - if reqType == 'events': - return handleEventsReq(params, dbCur) - elif reqType == 'info': - return handleInfoReq(params, dbCur) - elif reqType == 'sugg': - return handleSuggReq(params, dbCur) - return None - -# ========== For handling type=events ========== - -def handleEventsReq(params: dict[str, str], dbCur: sqlite3.Cursor) -> EventResponse | None: - """ Generates a response for a type=events request """ - # Get dates - dateRange = params['range'] if 'range' in params else '.' - if '.' not in dateRange: - print(f'INFO: Invalid date-range value {dateRange}', file=sys.stderr) - return None - try: - start, end = [reqParamToHistDate(s) for s in dateRange.split('.', maxsplit=1)] - except ValueError: - print(f'INFO: Invalid date-range value {dateRange}', file=sys.stderr) - return None - - # Get scale - if 'scale' not in params: - print('INFO: No scale provided', file=sys.stderr) - return None - try: - scale = int(params['scale']) - except ValueError: - print('INFO: Invalid scale value', file=sys.stderr) - return None - - # Get incl value - try: - incl = int(params['incl']) if 'incl' in params else None - except ValueError: - print('INFO: Invalid incl value', file=sys.stderr) - return None - - # Get result set limit - try: - resultLimit = int(params['limit']) if 'limit' in params else DEFAULT_REQ_EVENTS - except ValueError: - print(f'INFO: Invalid results limit {resultLimit}', file=sys.stderr) - return None - if resultLimit <= 0 or resultLimit > MAX_REQ_EVENTS: - print(f'INFO: Invalid results limit {resultLimit}', file=sys.stderr) - return None - - ctgs = params['ctgs'].split('.') if 'ctgs' in params else None - imgonly = 'imgonly' in params - - events = lookupEvents(start, end, scale, incl, resultLimit, ctgs, imgonly, dbCur) - unitCounts = lookupUnitCounts(start, end, scale, imgonly, dbCur) - - return EventResponse(events, unitCounts) - -def reqParamToHistDate(s: str): - """ Produces a HistDate from strings like '2010-10-3', '-8000', and '' (throws ValueError if invalid) """ - if not s: - return None - m = re.match(r'(-?\d+)(?:-(\d+)-(\d+))?', s) - if m is None: - raise ValueError('Invalid HistDate string') - if m.lastindex == 1: - return HistDate(None, int(m.group(1))) - else: - return HistDate(True, int(m.group(1)), int(m.group(2)), int(m.group(3))) - -def lookupEvents( - start: HistDate | None, end: HistDate | None, scale: int, incl: int | None, resultLimit: int, - ctgs: list[str] | None, imgonly: bool, dbCur: sqlite3.Cursor) -> list[HistEvent]: - """ Looks for events within a date range, in given scale, - restricted by event category, an optional particular inclusion, and a result limit """ - dispTable = 'event_disp' if not imgonly else 'img_disp' - query = \ - 'SELECT events.id, title, start, start_upper, end, end_upper, fmt, ctg, images.id, pop.pop FROM events' \ - f' INNER JOIN {dispTable} ON events.id = {dispTable}.id' \ - ' INNER JOIN pop ON events.id = pop.id' \ - ' LEFT JOIN event_imgs ON events.id = event_imgs.id' \ - ' LEFT JOIN images ON event_imgs.img_id = images.id' - constraints = [f'{dispTable}.scale = ?'] - params: list[str | int] = [scale] - - # Constrain by start/end - startUnit = dateToUnit(start, scale) if start is not None else None - endUnit = dateToUnit(end, scale) if end is not None else None - if startUnit is not None and startUnit == endUnit: - constraints.append(f'{dispTable}.unit = ?') - params.append(startUnit) - else: - if startUnit is not None: - constraints.append(f'{dispTable}.unit >= ?') - params.append(startUnit) - if endUnit is not None: - constraints.append(f'{dispTable}.unit < ?') - params.append(endUnit) - - # Constrain by event category - if ctgs is not None: - constraints.append('ctg IN (' + ','.join('?' * len(ctgs)) + ')') - params.extend(ctgs) - - # Add constraints to query - query2 = query - if constraints: - query2 += ' WHERE ' + ' AND '.join(constraints) - query2 += ' ORDER BY pop.pop DESC' - query2 += f' LIMIT {resultLimit}' - - # Run query - results: list[HistEvent] = [] - for row in dbCur.execute(query2, params): - results.append(eventEntryToResults(row)) - if incl is not None and incl == row[0]: - incl = None - - # Get any additional inclusion - if incl is not None: - row = dbCur.execute(query + ' WHERE events.id = ?', (incl,)).fetchone() - if row is not None: - if len(results) == resultLimit: - results.pop() - results.append(eventEntryToResults(row)) - - return results - -def eventEntryToResults( - row: tuple[int, str, int, int | None, int | None, int | None, int, str, int | None, int]) -> HistEvent: - eventId, title, start, startUpper, end, endUpper, fmt, ctg, imageId, pop = row - """ Helper for converting an 'events' db entry into an HistEvent object """ - # Convert dates - dateVals: list[int | None] = [start, startUpper, end, endUpper] - newDates: list[HistDate | None] = [None for n in dateVals] - for i, n in enumerate(dateVals): - if n is not None: - newDates[i] = dbDateToHistDate(n, fmt, i < 2) - - return HistEvent( - eventId, title, cast(HistDate, newDates[0]), newDates[1], newDates[2], newDates[3], ctg, imageId, pop) - -def lookupUnitCounts( - start: HistDate | None, end: HistDate | None, scale: int, - imgonly: bool, dbCur: sqlite3.Cursor) -> dict[int, int] | None: - """ Return list of units with counts given scale and a date range """ - # Build query - distTable = 'dist' if not imgonly else 'img_dist' - query = f'SELECT unit, count FROM {distTable} WHERE scale = ?' - params = [scale] - if start: - query += ' AND unit >= ?' - params.append(dateToUnit(start, scale)) - if end: - query += ' AND unit < ?' - params.append(dateToUnit(end, scale)) - query += ' ORDER BY unit ASC LIMIT ' + str(MAX_REQ_UNIT_COUNTS + 1) - - # Get results - unitCounts: dict[int, int] = {} - for unit, count in dbCur.execute(query, params): - unitCounts[unit] = count - return unitCounts if len(unitCounts) <= MAX_REQ_UNIT_COUNTS else None - -# ========== For handling type=info ========== - -def handleInfoReq(params: dict[str, str], dbCur: sqlite3.Cursor): - """ Generates a response for a type=info request """ - if 'event' not in params: - print('INFO: No \'event\' parameter for type=info request', file=sys.stderr) - return None - ctgs = params['ctgs'].split('.') if 'ctgs' in params else None - imgonly = 'imgonly' in params - return lookupEventInfo(params['event'], ctgs, imgonly, dbCur) - -def lookupEventInfo(eventTitle: str, ctgs: list[str] | None, imgonly: bool, dbCur: sqlite3.Cursor) -> EventInfo | None: - """ Look up an event with given title, and return a descriptive EventInfo """ - imgJoin = 'INNER JOIN' if imgonly else 'LEFT JOIN' - query = \ - 'SELECT events.id, title, start, start_upper, end, end_upper, fmt, ctg, images.id, pop.pop, ' \ - ' descs.desc, descs.wiki_id, ' \ - ' images.url, images.license, images.artist, images.credit FROM events' \ - ' INNER JOIN pop ON events.id = pop.id' \ - f' {imgJoin} event_imgs ON events.id = event_imgs.id' \ - f' {imgJoin} images ON event_imgs.img_id = images.id' \ - ' LEFT JOIN descs ON events.id = descs.id' \ - ' WHERE events.title = ? COLLATE NOCASE' - row = dbCur.execute(query, (eventTitle,)).fetchone() - if row is not None: - event = eventEntryToResults(row[:10]) - desc, wikiId, url, license, artist, credit = row[10:] - if ctgs is not None and event.ctg not in ctgs: - return None - return EventInfo(event, desc, wikiId, None if url is None else ImgInfo(url, license, artist, credit)) - else: - return None - -# ========== For handling type=sugg ========== - -def handleSuggReq(params: dict[str, str], dbCur: sqlite3.Cursor): - """ Generates a response for a type=sugg request """ - # Get search string - if 'input' not in params: - print('INFO: No \'input\' parameter for type=sugg request', file=sys.stderr) - return None - searchStr = params['input'] - if not searchStr: - print('INFO: Empty \'input\' parameter for type=sugg request', file=sys.stderr) - return None - - # Get result limit - try: - resultLimit = int(params['limit']) if 'limit' in params else DEFAULT_REQ_SUGGS - except ValueError: - print(f'INFO: Invalid suggestion limit {resultLimit}', file=sys.stderr) - return None - if resultLimit <= 0 or resultLimit > MAX_REQ_SUGGS: - print(f'INFO: Invalid suggestion limit {resultLimit}', file=sys.stderr) - return None - - ctgs = params['ctgs'].split('.') if 'ctgs' in params else None - imgonly = 'imgonly' in params - return lookupSuggs(searchStr, resultLimit, ctgs, imgonly, dbCur) - -def lookupSuggs( - searchStr: str, resultLimit: int, ctgs: list[str] | None, imgonly: bool, dbCur: sqlite3.Cursor) -> SuggResponse: - """ For a search string, returns a SuggResponse describing search suggestions """ - tempLimit = resultLimit + 1 # For determining if 'more suggestions exist' - query = 'SELECT title FROM events LEFT JOIN pop ON events.id = pop.id' \ - + (' INNER JOIN event_imgs ON events.id = event_imgs.id' if imgonly else '') \ - + ' WHERE title LIKE ?' - if ctgs is not None: - query += ' AND ctg IN (' + ','.join('?' * len(ctgs)) + ')' - query += f' ORDER BY pop.pop DESC LIMIT {tempLimit}' - suggs: list[str] = [] - - # Prefix search - params = [searchStr + '%'] + (ctgs if ctgs is not None else []) - for (title,) in dbCur.execute(query, params): - suggs.append(title) - - # If insufficient results, try substring search - if len(suggs) < tempLimit: - existing = set(suggs) - params = ['%' + searchStr + '%'] + (ctgs if ctgs is not None else []) - for (title,) in dbCur.execute(query, params): - if title not in existing: - suggs.append(title) - if len(suggs) == tempLimit: - break - - return SuggResponse(suggs[:resultLimit], len(suggs) > resultLimit) diff --git a/backend/server.py b/backend/server.py index 5c3904a..d03215c 100755 --- a/backend/server.py +++ b/backend/server.py @@ -8,7 +8,7 @@ from typing import Iterable import os from wsgiref import simple_server, util import mimetypes -from histplorer import application +from chrona import application import argparse parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) diff --git a/index.html b/index.html index d06306a..4ace086 100644 --- a/index.html +++ b/index.html @@ -3,7 +3,7 @@ - Histplorer + Chrona: Interactive Historical Timeline diff --git a/package.json b/package.json index 6c867f2..b135899 100644 --- a/package.json +++ b/package.json @@ -1,5 +1,5 @@ { - "name": "histplorer", + "name": "chrona", "private": true, "version": "0.1.0", "description": "An interactive historical timeline", diff --git a/prebuild.sh b/prebuild.sh new file mode 100755 index 0000000..9731276 --- /dev/null +++ b/prebuild.sh @@ -0,0 +1,7 @@ +#!/bin/bash +set -e + +sed -i -e "s|base: .*,|base: '/chrona/',|" vite.config.ts +sed -i -e "s|SERVER_DATA_URL = .*|SERVER_DATA_URL = (new URL(window.location.href)).origin + '/chrona/data/'|" \ + -e "s|SERVER_IMG_PATH = .*|SERVER_IMG_PATH = '/img/chrona/'|" src/lib.ts +sed -i -e 's|DB_FILE = .*|DB_FILE = "/usr/local/www/db/chrona.db"|' backend/chrona.py diff --git a/src/README.md b/src/README.md index 409e796..5ef7892 100644 --- a/src/README.md +++ b/src/README.md @@ -2,6 +2,21 @@ - **main.ts**: Included by ../index.html. Creates the main Vue component. - **App.vue**: The main Vue component. - **components**: - - **TestComponent.vue**: For testing. -- **index.css**: Included by main.ts. Provides Tailwind's CSS classes. + - **TimeLine.vue**: Displays an interactive timeline. + - **BaseLine.vue**: Displays a timeline overview. + - **InfoModal.vue**: Modal displaying event info. + - **SearchModal.vue**: Modal providing a search bar. + - **SettingsModal.vue**: Modal displaying configurable settings. + - **HelpModal.vue**: Modal displaying help info. + - **LoadingModal.vue**: Displays a loading indicator. + - **SButton.vue**: Simple button component. + - **IconButton.vue**: Simple button component containing an SVG icon. + - **SCollapsible.vue**: Simple collapsible-content component. + - **icon**: Contains components that display SVG icons. +- **store.ts**: App global storage. +- **lib.ts**: Holds project-wide globals. +- **util.ts**: Holds utility functions. +- **rbtree.ts**: Red-black tree implementation. +- **index.css**: Included by main.ts. Provides Tailwind's CSS classes. - **vite-env.d.ts**: From Vite's template files. +- **global.d.ts**: Temporary typescript overrides. diff --git a/src/lib.ts b/src/lib.ts index 51bd2a4..051a449 100644 --- a/src/lib.ts +++ b/src/lib.ts @@ -1,5 +1,5 @@ /* - * Common project globals + * Project-wide globals */ import {moduloPositive, intToOrdinal, getNumTrailingZeros} from './util'; -- cgit v1.2.3