ARK - New pages [en]

Brother A3

2018-08-22T14:34:26Z

John Layt: /* Scanning Permatrace */

= Brother A3 Printer / Scanner =

LP owns a number of Brother A3 Printer / Scanners, primarily for scanning Permatrace, but also for printing large A3 plans and drawings, or colour documents where an office does not have a colour laser printer.

NOTE: These printers are Ink Jet printers, which means the inks are not archivally stable and so cannot be used for printing Recording Sheets and any other documents to be deposited as part of the site archive. You must use a laser printer for these purposes.

== Setup ==

The printers should be setup to connect via the office network, although sometimes they may be directly connected via USB to a machine. If you need any assistance with this please contact the Sysadmins.

Apple macOS will automatically detect the printer and scanner once they are connected to the network and they can be immediately used from the macOS print dialog or Image Capture app (under the Shared section, click on more to display). They will usually be listed with a model number like ''Brother MFC J6930DW''.

If you need to use some advanced features, or do large volumes of Permatrace scanning, then you will need to install the official Brother drivers and app. The drivers can be found in the Library/Software/Printer Scanner Drivers/Brother A3/ folder, or downloaded from http://support.brother.com/g/b/countrytop.aspx?c=gb&lang=en. If you need Admin rights to install the drivers, please talk to your managing partner or the Sysadmins.

== Scanning Permatrace ==

You can use the standard Apple Image Capture app to scan permatrace using the Automatic Document Feeder (ADF), but this app will only allow you to do so at A3 paper size. While the image resolution will be correct you will have to manually crop the image to permatrace size afterwards.

The Brother Control Centre app enables you to scan custom page sizes using the ADF. You can find the app in the Finder under Applications / Brother / Control Centre. If it is not installed, see the Setup section for installation instructions.

Run the Control Centre, choose the printer ''Model'' you want to use, and click on the ''Custom Scan'' tab to see if the ''Permatrace'' option has already been added:

[[File:Brother_scan_custom.png]]

Load the scanner ADF with up to 30 sheets of permatrace. Click on the ''Permatrace'' button to get the ''Save to File'' dialog:

[[File:Brother_scan_file.png]]

Change the file name prefix to the LP standard format for the site and drawing type you are scanning. This will usually be something like cxt_STE18_ for context drawings for site STE18 (see the LP Data Standards for further guidance).

Use the ''Browse'' button to select the folder to save the scans to. We recommend saving to a local folder and not the server as it will be faster and easier to manage the files afterwards.

Ensure all other settings are as displayed above.

Click ''Start Scanning'' button which will show the ''Brother TWAIN'' dialog to set the image scan details. This will default to the last used values, you should not change these once set the first time:

[[File:Brother_scan_settings.png]]

Once you click start the permatrace will start feeding through the ADF and saving to your local folder. You need to watch the ADF carefully as dirty permatrace misfeeds easily. If it does misfeed you have to unload all the permatrace, close the scan dialog, reload the permatrace, and start the scan again. If the ADF clicks on a misfeed you have enough time

To create the Permatrace option click on the Configuration button and choose the Custom1 option to edit:

This will show a Config dialog which needs to be set up as follows:

Click OK to save and you'll have the Permatrace option.

You'll only need to fill these in the first time, afterwards it will remember them once you click Start.

ARK2/View

2018-03-19T15:47:58Z

John Layt: /* Tables */

= View =

ARK2 provides a table driven configuration of Views on Data Schemas and Items, such as Pages, Forms and Tables. View Elements can be nested to build up complex layouts.

== View Elements ==

View elements are either Basic Elements, or Compound Elements made up of other Basic and/or Compound Elements.

Basic Elements
* Widget - An object embedded in a page, e.g. a form element, text element, button, etc
* Field - A widget that is linked to a specific Attribute in a Schema and inherits its behaviour from that Attribute, e.g. an Actor's name is a text form field with defined size constraints
* Nav - A widget that is used for site navigation, e.g. a clickable link with or without an icon

Compound Elements
* Page - A layout of other elements in a page, with standard elements such as header, sidebar, footer, and content
* Grid - A layout of other elements in rows and columns, both basic and compound
* Form - An HTML5 Form
* Table - An HTML5 Table
* Tree - Currently not implemented, a layout of other elements in a hierarchical tree, e.g. a navigation menu

== View Tables / Classes ==

ark_view_cell - ARK\View\Cell
ark_view_element - ARK\View\Element
ark_view_field - ARK\View\Field
ark_view_form - ARK\View\Form
ark_view_group - ARK\View\Grid
ark_view_nav - ARK\View\Nav
ark_view_page - ARK\View\Page
ark_view_table - ARK\View\Table
ark_view_tree - ARK\View\Tree
ark_view_type - ARK\View\Type
ark_view_widget - ARK\View\Widget

== Tables ==

Tables are a Compound Element defined by the ARK\View\Table class, and can only embed ARK\View\Field or ARK\View\Widget Basic Elements. They cannot embed other Compound Elements. When a Table embeds Field Elements, then all Fields must belong to a single Schema, i.e. a Table embeds a Schema, with each row embedding an Item that is an instance of that Schema.

The Table Element is defined in the ark_view_table table, which is mapped by Doctrine to the ARK\View\Table class. The definition seeks to abstract the HTML5 Table definition and the currently chosen JavaScript Table widget in a generic way to avoid implementation specifics. The Twig or JavaScript frontends are expected to translate this generic definition into the specific implementation being used.

{| class="wikitable"

|+ Table Definition

! Database Field
! Class Method
! Datatype
! Description

|-
| element
| id()
| string
| The unique ID of the Table element
|-
| name
| name()
| string
|
|-
|
| type()
| ARK\View\Type
|
|-
| mode
| mode()
| boolean
|
|-
| caption
| showCaption()
| boolean
| Whether to show the HTML5 table caption element
|-
| header
| showHeader()
| boolean
| Whether to show the HTML5 table header element
|-
| footer
| showFooter()
| boolean
| Whether to show the HTML5 table footer element
|-
| sortable
| isSortable()
| boolean
| Whether the table rows can be sorted
|-
| searchable
| isSearchable()
| boolean
| Whether the table data is searchable
|-
| list
| enableListView()
| boolean
| Whether the table list view is enabled
|-
| detail
| enableDetailView()
| boolean
| Whether the table detail view is enabled
|-
| expand
| enableExpandListView()
| boolean
| Whether the table expand view is enabled (only if list and detail views are enabled)
|-
| card
| enableCardView()
| boolean
| Whether the table card view is enabled
|-
| thumb
| enableThumbnailView()
| boolean
| Whether the table thumbnail view is enabled
|-
| view
| defaultView()
| string
| Which table view is the default view, choice of list, card, or thumb
|-
| image
| image()
| string
| The data field to use for images, i.e. thumbnails
|-
| export
| canExportData()
| boolean
| Whether the table data can be exported
|-
| columns
| canChooseColumns()
| boolean
| Whether the table columns displayed can be selected
|-
| pagination
| pagination(), pageSize()
| boolean, integer
| The number of rows per page, defaults to 10
|-
| selection
| selection()
| string
| Whether data rows can be selected
|-
| keyword
| keyword()
| string, ARK\Translation\Keyword
| The translation keyword used for the table
|-
| classes
| classes()
| string
| Any extra CSS classes to add to the table
|-
| template
| template()
| string
| The TWIG template to use instead of the default table template
|-
| url
| dataUrl()
| string
| The server-side pagination URL
|-

|}

ARKmatrix

2018-03-17T14:40:37Z

John Layt: /* Harris Matrix Software */

= Harris Matrix Software =

This page summarises L - P : Archaeology's ongoing work on Harris Matrix software.

* Our ARK Matrix software: https://gitlab.com/arklab/ArkMatrix
* Our survey of the State of the Art in 2015: http://archaeologic.al/wiki/Harris_Matrix
* The Matrix Reloaded - CAA UK 2016 presentation by John Layt of L - P : Archaeology https://www.youtube.com/watch?v=geLD7Fo6crU

== ARK Matrix Software ==

High-level sketch of docs - a Work-in-Progress!

=== Strat Units ===

Unit Attributes
* Site Code
* Unit ID
* Label
* Class [Context, Subgroup, Group, Landuse]
* Type [Deposit, Fill, Cut, Masonry, Skeleton, Timber]
* Status [Allocated, Assigned, Void]
* Aggregate Unit
* Phase
* Period

=== Strat Relations ===

* Above / Below (directed graph)
* Same-As (undirected graph)
* Contemporary With (undirected graph) [REMOVE?]

=== Rules / Heuristics ===

Any Relations:
* Cannot be between same Unit
* Both Units must be same Class

Above/Below Relations:
* Adding relation should not create cycles (no intersection between successors and predecessors)

Same-As Relations:
* Both Units must be Context class
* Adding relation should not create cycles (no intersection between successors and predecessors)
* Both Units must have the same Aggregate Unit if set
* Both Units must have the same Period / Phase if set

=== Processing ===

Load

Validate
* Validate Units
* Validate Above/Below relations
* Validate Same-as Units

Resolve
* For each Same-As relationship add all Above/Below relations to each Unit

Validate

Reduce
* Remove all redundant Above/Below relations (transitive reduction)

Aggregate
* Validate all Context Units have an Aggregate Unit
* For each Context Above/Below relation where the Aggregate Unit is different between the Above and Below Units, add an Above/Below relation between the Aggregate Units to the Subgroup matrix
* Reduce the Subgroup matrix
* Validate all Subgroup Units have an Aggregate Unit
* For each Subgroup Above/Below relation where the Aggregate Unit is different between the Above and Below Units, add an Above/Below relation between the Aggregate Units to the Group matrix
* Reduce the Group matrix

== Development Plans ==

* https://github.com/anvaka/graph-drawing-libraries
* https://github.com/ArsMasiuk/qvge

ARK2/The ARK Way

2017-12-16T12:25:03Z

John Layt: /* Inspiration */

== Inspiration ==

The architecture, design and implementation of ARK2 was deeply inspired by [http://www.phptherightway.com/ PHP The Right Way], a manifesto for developing modern PHP web applications. Following in that spirit, this page sets out how ARK2 implements the recommendations of [http://www.phptherightway.com/ PHP The Right Way], as well as [https://symfony.com/doc/current/best_practices/index.html Symfony Best Practices], and leavened with our own decades of development experience outside the web world.

This page will broadly follow the organisation of PHP The Right Way, addressing each recommendation in turn

ARK2/Maps

2017-11-30T12:10:44Z

John Layt: /* Configuration */

= Maps =

ARK2 supports the display of maps and mapping layers using OpenLayers4. Maps and layers can be dynamically defined in the config database and then linked to a defined page view. Much of the config closely reflects the [http://openlayers.org/en/latest/apidoc/ OpenLayers4 API] as this is the map library used by the ARK2 frontend.

== Configuration ==

The following database tables can be configured to define an OL4 map.

Define available [http://openlayers.org/en/latest/apidoc/ol.source.html OL4 Sources] in ark_map_source.

{| class="wikitable" style="margin: 0; border: 1px solid grey; text-align:left" cellpadding="4" cellspacing="2"
! colspan="6" | ark_map_source
|-
! Field || Type || Size || Required || Description || Example
|-
| source || varchar || 30 || PK || The key for the source || bing
|-
| type || varchar || 30 || Y || The type of the source (raster, vector) || raster
|-
| subtype || varchar || 30 || Y || The subtype of the source (image, tile, vector) || tile
|-
| format || varchar || 30 || Y || The format of the source (wms, gml, etc) || bing
|-
| view_class || varchar || 100 || Y || The [http://openlayers.org/en/latest/apidoc/ol.source.html OL4 Class] of the source || BingMaps
|-
| keyword || varchar || 100 || Y || The ARK translation keyword || map.source.bing
|-
| ticket || varchar || 100 || N || The API auth ticket for the source ||
|-
| ticket_expiry || datetime || || N || The expiry date for the API auth ticket ||
|-
| options || varchar || 4000 || N || The OL4 Options to pass to the Source class in JSON format ||
|-
|}

Define available [http://openlayers.org/en/latest/apidoc/ol.layer.html OL4 Layers] for each Source in ark_map_layer.

{| class="wikitable" style="margin: 0; border: 1px solid grey; text-align:left" cellpadding="4" cellspacing="2"
! colspan="6" | ark_map_layer
|-
! Field || Type || Size || Required || Description || Example
|-
| source || varchar || 30 || PK || The source key from ark_map_source || bing
|-
| layer || varchar || 30 || PK || The layer key || aerial
|-
| source_name || varchar || 50 || Y || The key of the layer as defined by the source provider || Aerial
|-
| keyword || varchar || 100 || Y || The ARK translation keyword || map.layer.bing.aerial
|-
| url || varchar || 2000 || N || The url of the layer ||
|-
| options || varchar || 4000 || N || The OL4 Options to pass to the Layer class in JSON format ||
|-
| parameters || varchar || 4000 || N || The layer parameters to pass to the source provider ||
|-
|}

Define available [http://openlayers.org/en/latest/apidoc/ol.Map.html OL4 Map] views in ark_map.

{| class="wikitable" style="margin: 0; border: 1px solid grey; text-align:left" cellpadding="4" cellspacing="2"
! colspan="6" | ark_map_source
|-
! Field || Type || Size || Required || Description || Example
|-
| map || varchar || 30 || PK || The key for the map || ark_map_main
|-
| draggable || boolean || || Y || If the map view is draggable ||
|-
| clickable || boolean || || Y || If the map view is clickable ||
|-
| zoomable || boolean || || Y || If the map view is zoomable ||
|-
| zoom || integer || || Y || The default zoom level || 7
|-
| min_zoom || integer || || N || The minimum zoom level || 6
|-
| max_zoom || integer || || N || The maximum zoom level || 16
|-
| srid || integer || || N || The SRID of the map projection || 4512
|-
| center || varchar || 50 || Y || The default center point of the map view in WKT format || POINT (1155972 7580813)
|-
| extent || varchar || 100 || N || The maximum extent of the map view in WKT format || MULTIPOINT (831000 7230000, 1750000 7950000)
|-
| keyword || varchar || 100 || N || The ARK translation keyword || ark.map.main
|-
| options || varchar || 4000 || N || The OL4 Options to pass to the Map class in JSON format ||
|-
|}

Define the Source Layers displayed in a Map View in the ark_map_legend table.

{| class="wikitable" style="margin: 0; border: 1px solid grey; text-align:left" cellpadding="4" cellspacing="2"
! colspan="6" | ark_map_layer
|-
! Field || Type || Size || Required || Description || Example
|-
| map || varchar || 30 || PK || The map key from ark_map || ark_map_main
|-
| source || varchar || 30 || PK || The source key from ark_map_layer || bing
|-
| layer || varchar || 30 || PK || The layer key from ark_map_layer || aerial
|-
| seq || integer || || Y || The sequence of the layer in the legend || 1
|-
| is_default || boolean || || Y || If the view layer is the default ||
|-
| visible || boolean || || Y || If the view layer is visible by default ||
|-
| enabled || boolean || || Y || If the view layer is enabled ||
|-
| keyword || varchar || 100 || Y || The ARK translation keyword || map.layer.bing.aerial
|-
| options || varchar || 4000 || N || The OL4 Options to pass to the Layer class in JSON format ||
|-
|}

Available config

2017-11-27T16:10:16Z

Mike: /* ARK Empty */

With the ARK1.2 release there are several available default setups or "configs" for ARK. When installing ARK you must choose the right one for your project.

===Single Context===
This is L - P : Archaeology's favourite config, it is a full implementation of the MoL recording system including modules for Contexts, Sections, Plans, Registered Finds, Samples, Site Photographs, Timber, 'Trenches, Areas, Etc' (TAE), Subgroups, Groups and Landuses.

===Basic Context===
This is a simple version of the single context recording system which has modules for Contexts, Plans, Sections, Registered Finds and Site Photos.

===ARK Landscape===
This config is good for landscape studies it has modules for Locations, Excavations, Finds, Catalogued Finds, Illustrations and Photos.

===ARK Stratigraphic Unit===
This config is used by some mediterranean projects, it has modules for Contexts, Site Photos, Plans, Geophotos, Architectural Elements, Special Finds and Coins

===ARK Empty===
ARK is endlessly customisable, the empty config has only the required addressbook module, allowing you to create your own config with modules from the other configs, or by creating them yourself. check out the [http://ark.lparchaeology.com/wiki/index.php/Mod_settings.php Mod Settings] page to see how.

ARK2/Community

2017-05-27T11:46:10Z

John Layt:

A development and support community will be required for both the Project and the Service.

* User support
* Sysadmins support
* Developers support

An online chat service that integrates with an issue tracker, project manager, wiki, and code repository will be needed for developers. A chat, wiki, and forum will be needed for user support. It makes sense to use a single solution for both, either under a single brand or separate brands. The Service will need to be able to keep some Projects, Clients, and Tasks private, with the rest remaining open.

Needed Features:
* Guest or OAuth
* Chat: web and app
* Wiki
* GitHub integration
* CI system
* Bots

Chat options:
* Basecamp / Campfire - More customer oriented than coder oriented, costs $1000 p/a
* Gitter - Integrated with GitHub and common developer services, free but not open, dodgy security, login only via GitHub or Twitter account
* Atlassian / HipChat - Lots of integration, free hosted/paid, not open
* RocketChat - Free and Open, self-serve, host very expensive, fewer integrations, but screenshare and videochat are free.
* Slack
* Flock https://flock.com/uk/pricing/
* Mattermost https://about.mattermost.com/features/
* Matrix / Riot https://matrix.org/ / https://about.riot.im/need-help/
* Discourse https://www.discourse.org/features
* https://www.getopensocial.com/
* https://www.oxwall.com/
* https://www.humhub.org/en

Project Management Options:
* https://huboard.com/pricing
* https://waffle.io/pricing/
* https://www.zenhub.com/pricing
* https://overv.io/

Help Ticket:
* https://github.com/ladybirdweb/faveo-helpdesk

Other:
* https://secure.phabricator.com/applications/
* https://www.gitbook.com/pricing
* https://zapier.com/

ARK2/Install

2017-01-09T18:08:35Z

John Layt: /* Debian Stable (Stretch) */

= Install =

== Git / Github ==

Development is hosted on [https://gitlab.com/arklab GitLab], so you will need a free GitLab account to contribute code.

If you are new to Git, you may find a desktop application easier to use such as [https://www.sourcetreeapp.com/ SourceTree] and [https://www.gitkraken.com/ GitKraken], or the support built-in to your IDE like Atom.

== Environment ==

To develop ARK requires the following tools to be installed:
* [https://git-scm.com/ Git]
* [https://getcomposer.org/ Composer]
* [https://nodejs.org/en/ Node.js] (for frontend/theme development only)

You will also require PHP, a web server (Apache/PHP), a database server (MySQL/PostreSQL/SQLite), a web browser (Chrome/Firefox), and a an API client (Postman).

The following tools are recommend to adhere to ARK code quality standards:
* php-cs-fixer
* PHP CodeSniffer

=== macOS ===

On macOS, while you can install the requirements via standalone packages such as [https://www.mamp.info/en/ MAMP], we recommend using HomeBrew as it makes installing and updating all the tools used easier. Using macOS is only recommended for local hosting or development purposes only, internet-connected production sites should be run on a properly supported server installation, such as Debian Stretch.

* Install XCode from the App Store and run it to accept the license, or from the command line: <pre>sudo xcodebuild -license accept</pre>
* Check you have the Command Line Tools (including Git) installed and enabled: <pre>sudo xcode-select --install</pre>
* Download and install [http://brew.sh/ HomeBrew]: <pre>/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"</pre>
* Modify your ~/.profile file to add brew packages to your $PATH and to enable Git completion
<pre> export PATH=/usr/local/bin:/usr/local/sbin:$PATH
source /Library/Developer/CommandLineTools/usr/share/git-core/git-completion.bash
source /Library/Developer/CommandLineTools/usr/share/git-core/git-prompt.sh
GIT_PS1_SHOWDIRTYSTATE=true
PS1='\u@\h \W$(__git_ps1 " (%s)") \$ '
</pre>
* Source your modified ~/.profile file to start using it: <pre>source ~/.profile</pre>
* Install the required packages from Brew:
<pre>
brew install httpd mariadb imagemagick icu4c
brew install php@7.4
pecl install apcu imagick xdebug
brew install composer php-code-sniffer php-cs-fixer@2 phpdocumentor phplint phpmyadmin
brew install node@14
</pre>
* Modify /usr/local/etc/httpd/httpd.conf to enable PHP-FPM, aliases, vhosts and rewrite by uncommenting the following lines:
<pre>
LoadModule proxy_module lib/httpd/modules/mod_proxy.so
LoadModule proxy_fcgi_module lib/httpd/modules/mod_proxy_fcgi.so
LoadModule vhost_alias_module lib/httpd/modules/mod_vhost_alias.so
LoadModule alias_module lib/httpd/modules/mod_alias.so
LoadModule rewrite_module lib/httpd/modules/mod_rewrite.so
</pre>
* Add the following after the DocumentRoot section to enable PhpMyAdmin:
<pre>
Alias /phpmyadmin /usr/local/share/phpmyadmin
<Directory /usr/local/share/phpmyadmin/>
Options Indexes FollowSymLinks MultiViews
AllowOverride All
Require all granted
</Directory>
</pre>
* Modify the DirectoryIndex section to appear as follows:
<pre>
#
# DirectoryIndex: sets the file that Apache will serve if a directory
# is requested.
#
<IfModule dir_module>
DirectoryIndex index.html index.php
</IfModule>

<IfModule proxy_fcgi_module>
<FilesMatch ".+\.ph(ar|p|tml)$">
SetHandler "proxy:fcgi://127.0.0.1:9000"
</FilesMatch>
</IfModule>
</pre>
* Start the required services:
<pre>
brew services start mariadb
brew services start php@7.4
brew services start httpd
</pre>
* Point your browser to localhost and you should see the default Apache page
* Point your browser to localhost/phpmyadmin and sign in to confirm that PHP and MariaDB are running correctly

=== Debian ===

Since Debian Stretch, Debian has enabled the parallel install of multiple PHP versions using PHP-FM. However each release of Debian ships with a different default version and only that version is officially supported in that release. In order to install other versions such as PHP 5.6 and 7.4 you need to use the packages made available by the Debian Maintainer in his personal development repo. Check your version of Debian to determine what you need. LP Archaeology use this solution for our servers and we strongly recommend you use this method for both simplicity and reliability.

Instructions for NodeJS: https://nodejs.org/en/download/package-manager/#debian-and-ubuntu-based-linux-distributions

Instructions for adding PHP repo: https://packages.sury.org/php/README.txt. See also https://github.com/oerdnj/deb.sury.org/wiki/Managing-Multiple-Versions

Summary of install commands for PHP 7.4 current requirement:

sudo apt-get install build-essential git apache2 mariadb-client mariadb-server

sudo apt-get install php7.4 php7.4-apcu php7.4-apcu-bc php7.4-bcmath php7.4-bz2 php7.4-cli php7.4-common php7.4-fpm php7.4-gd \
php7.4-geoip php7.4-imagick php7.4-intl php7.4-json php7.4-mbstring php7.4-mysql php7.4-opcache php7.1-readline php7.4-sqlite3 \
php7.4-tidy php7.4-uuid php7.4-yaml php7.4-xml php7.4-zip phpmyadmin php-pear

curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -
sudo apt-get install -y nodejs

curl -sS https://getcomposer.org/installer -o composer-setup.php
php composer-setup.php --install-dir=/usr/local/bin --filename=composer

Enabled FPM

sudo service php7.4-fpm start
sudo systemctl enable php7.4-fpm
sudo a2enconf php7.4-fpm
sudo a2enmod proxy_fcgi rewrite

Once ARK is installed (see below), configure Apache to serve the site, using either alias or symlink.

sudo vim /etc/apache2/sites-available/mysite.conf

Using a simple alias on a subdirectory:

<VirtualHost *:80>
ServerName localhost
ServerAdmin sysadmin@localhost
DirectoryIndex index.html index.php
LogLevel debug
Alias "/mysite"
<Directory "/srv/www/lib/ark2/sites/mysite/web">
Options FollowSymLinks
Require all granted
AllowOverride None
AcceptPathInfo On
RewriteEngine On
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^(.*)$ /index.php [QSA,L]
</Directory>
</VirtualHost>

Using a vhost on a subdomain

<VirtualHost *:80>
ServerName mysite.example.com
ServerAdmin admin@example.com
DocumentRoot /srv/www/lib/ark2/sites/mysite/web
DirectoryIndex index.html index.php
LogLevel warn
ErrorLog /path/to/logs/<site>/error.log
CustomLog /path/to/logs/<site>/access.log combined
<Directory "/srv/www/lib/ark2/sites/mysite/web">
Options FollowSymLinks
Require all granted
AllowOverride None
AcceptPathInfo On
RewriteEngine On
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^(.*)$ /index.php [QSA,L]
</Directory>
</VirtualHost>

sudo a2ensite mysite

sudo apache2ctl -t

sudo systemctl reload apache2

sudo apache2ctl -S

== Install ==

To install using git, create the folder where the library code will be located, then clone the ARK2 repository.

For example, for Debian stretch:

mkdir -p /srv/www/lib
cd /srv/www/lib
git clone https://gitlab.com/arklab/arkserver.git

Alternatively download and unzip the file from https://gitlab.com/arklab/arkserver

Then run composer to install the required PHP libraries:

cd arkserver/
composer install
<may need github token...>

Run the Sysadmin console to check your install:

./bin/sysadmin
./bin/sysadmin system:about

Check the database server config file is correct for your install:

less sites/servers.json

If you need to add different server details:

./bin/sysadmin database:server:add

To create a new site:

./bin/sysadmin site:create <site>
./sites/<site>/bin/console

Now edit the site config files to reflect your site settings. Most settings will have the correct default, but some may need tweaking:

vim sites/<site>/config/site.json

If using vhosts, see above for instructions.

If not using vhosts, for local development, edit your Apache config to alias the site folder:

Alias /<site> /path/to/ark2/sites/<site>/web
<Directory /path/to/ark2/sites/<mysite>/web>
Options Indexes FollowSymLinks MultiViews
AllowOverride All
Require all granted
</Directory>

http://localhost/<site>

To rebuild the frontend
cd build
npm install
./build frontend:all <frontend>

ARK2/Console

2017-01-04T10:07:50Z

John Layt: /* Console */

= Console =

ARK2 uses the [https://symfony.com/doc/current/components/console.html Symfony Console] to provide three custom command-line consoles for maintaining ARK:

* The System Console - A console to assist in managing an ARK system install (./bin/sysadmin)
* The Site Console - A console to assist in managing an ARK site in a system install (./sites/<site>/bin/console)
* The Build Console - A console to assist in developing ARK and ARK front-ends (./build/build)

The key difference between the System and Site consoles is the System console does not load a specific site configuration when it runs. This allows it to perform maintenance on the whole system or on an individual site even when an individual site configuration is unable to be loaded. The Site Console needs to load a valid site configuration to allow it to perform more advanced site-specific maintenance.

The Build Console is for developer use only and is documented under the Frontend Build documentation.

== System Console ==

The System Console can be run from the command line from the root install folder:

./bin/sysadmin

Running the console without and parameters displays a help message and the list of available commands:

<pre>
ARK System Admin Console 1.9.80

Usage:
command [options] [arguments]

Options:
-h, --help Display this help message
-q, --quiet Do not output any message
-V, --version Display this application version
--ansi Force ANSI output
--no-ansi Disable ANSI output
-n, --no-interaction Do not ask any interactive question
-v|vv|vvv, --verbose Increase the verbosity of messages: 1 for normal output, 2 for more verbose output and 3 for debug

Available commands:
help Displays help for a command
list Lists commands
cache
cache:clear Clear the system cache
database
database:clone Clone an existing database
database:drop:tables Drop all tables in a database
database:import Import an SQL file into a database
database:server:add Add a new database server
database:truncate Truncate all tables in a database
security
security:check Checks security issues in your project dependencies
site
site:create Create a new site
site:frontend Install or replace a frontend. WARNING: Will delete the old frontend!
site:migrate:info Analyse the migration mapping for an ARK 1 site
site:migrate:load Migrate an ARK 1.2 site
site:migrate:map Create the migration mapping for an ARK 1 site
system
system:about Display information about the system
</pre>

== Site Console ==

The System Console can be run from the command line from the root install folder:

./bin/sysadmin

Running the console without and parameters displays a help message and the list of available commands:

<pre>
ARK Site Admin Console 1.9.80

Usage:
command [options] [arguments]

Options:
-h, --help Display this help message
-q, --quiet Do not output any message
-V, --version Display this application version
--ansi Force ANSI output
--no-ansi Disable ANSI output
-n, --no-interaction Do not ask any interactive question
-v|vv|vvv, --verbose Increase the verbosity of messages: 1 for normal output, 2 for more verbose output and 3 for debug

Available commands:
help Displays help for a command
list Lists commands
cache
cache:clear Clear the system cache
database
database:drop:tables Drop all tables in a database
database:import Import an SQL file into a database
database:truncate Truncate all tables in a database
orm
orm:entity:generate Generate ORM Entities for Custom ARK Modules
route
route:dump Dump configured route(s)
site
site:about Display information about the site
spatial
spatial:rebuild Rebuild the Spatial Index
translation
translation:add Add or Set a translation.
user
user:create Create a new user
user:delete Delete a user
user:disable Disable a user
user:enable Enable a user
user:list List all users.
user:password:reset Send user a password reset email
user:password:set Set the password for a user
user:role:add Add a user role
user:role:delete Delete a user role
user:verify Verify a user's email
view
view:page:dump Dump a View Page
</pre>

ARK2/Templates

2017-01-02T13:37:15Z

John Layt: /* Extensions */

= Templates =

ARK2 uses the TWIG templating engine to define the Front-end web views.

See the [http://twig.sensiolabs.org/documentation Twig documentation] on general use of Twig.

See the [http://silex.sensiolabs.org/doc/providers/twig.html Silex documentation] for using TWIG with the Silex Framework.

See the [http://symfony.com/doc/current/templating.html Symfony documentation] for using TWIG with the Symfony Components

== Extensions ==

TWIG can easily be extended with custom extensions to the template language. ARK2 ships with a number of common extensions, as well as some ARK-specific extensions.

* ARK:
** service: A global variable to access the ARK Service object.
** translate: An extension to the standard Symfony trans option to add support for ARK Translation Roles, see [[ARK2/Localization#Translation|Translation]].
** translatechoice: An extension to the standard Symfony transchoice option to add support for ARK Translation Roles, see [[ARK2/Localization#Translation|Translation]].

* [http://silex.sensiolabs.org/doc/providers/twig.html Silex/Symfony Extensions] , in particular:
** [http://symfony.com/doc/current/book/translation.html#twig-templates Translation]
** [http://symfony.com/doc/current/reference/forms/twig_reference.html Forms]
** [http://symfony.com/doc/current/book/security.html#access-control-in-templates Security]
** Asset
** Global variables for: app, request, session, user, debug

* [http://twig.sensiolabs.org/doc/extensions/index.html Official Twig Extensions], in particular:
** Text: Provides useful filters for text manipulation;
** Intl: Adds a filter for localization of DateTime objects, numbers and currency;
** Array: Provides useful filters for array manipulation;
** Date: Adds a filter for rendering the difference between dates.

ARK2/Workflow

2016-12-30T11:19:04Z

John Layt: Created page with "= Workflow = * https://github.com/Sylius/SyliusFlowBundle * https://www.w3.org/TR/activitystreams-core/"

= Workflow =

* https://github.com/Sylius/SyliusFlowBundle
* https://www.w3.org/TR/activitystreams-core/

ARK2/Cache

2016-12-27T15:27:11Z

John Layt: /* Implementation */

= Cache =

Caching can greatly improve performance of ARK, especially for configuration and metadata, but also the data and database access. ARK provides a flexible caching system based on Doctrine Cache, Symphony Cache, and the PSR-6 Caching standard, and supporting filesystem, APCu, or Redis caches (others are possible but not officially supported).

By default, ARK will use APCu for caching if your server has APCu installed, otherwise a filesystem cache will be used. For high-volume sites and APIs, Redis is recommended using either Predis or PHPRedis. Redis can also be used as a Spatial cache to speed up spatial searches.

== Configuration ==

If caching is not explicitly configured in the site.json file, then ARK will check if APCu is available, and if not will fall back to the filesystem. To fractionally improve performance, you can use site.json to explicitly set 'apc' or 'file'.

If using Redis you need to explicitly set site.json to use 'redis' and the url of the Redis instance.

If Debug mode is enabled, then the cache is disabled.

== Implementation ==

ARK uses the Symfony Cache component for a PSR-6 compliant implementation. This also provides a bridge for the Doctrine ORM to use for caching.

Two caches are provided, the System and Application caches:
* System: An APCu cache primarily for PHP compilable items or smaller static items such as config and metadata.
* App: A cache for application data.

These caches are accessed via the container:
* $app['cache.app'] or $app['cache']
* $app['cache.system']
* $app['cache.orm.app']
* $app['cache.orm.system']

The global Service object also provides access to the cache.

== Commands ==

* cache::warmup - Warms-up the System cache
* cache::clear - Clears the System cache
* cache::refresh - Clears and warms-up the System cache
* doctrine cache commands?

Any cachable component should provide HTTP Kernel CacheWarmer and CacheClearer implementations and register these with the cache provider.

ARK2/Vocabulary

2016-12-19T14:13:30Z

John Layt: /* Classes */

= Vocabulary =

ARK1 supported Controlled Lists of Attributes. In ARK2 we support other types of Controlled Vocabularies including Taxonomies and Thesauri, in compliance with ANSI/NISO standard [http://www.niso.org/publications/ansiniso-z3919-2005-r2010-guidelines-construction-format-and-management-monolingual Z39.19-2005 'Guidelines for the Construction, Format, and Management of Monolingual Controlled Vocabularies'].

== Definitions ==

* Concept - What the Vocabulary is about, such as Country, Period, Material, etc
* Term - A valid value within the Vocabulary, such as Denmark, Viking, Gold, etc
* Type - The Type of Controlled Vocabulary which controls what relationships the Vocabulary has between Terms (Equivalence, Hierarchy, Association)
* Equivalence - Where two Terms are considered the same
* Hierarchy - Where two Terms have a parent/child relationship, usually called broader/narrower
* Association - Where two Terms have a relationship that is not Equivalence nor Hierarchy, i.e. related terms, order, etc
* List - A simple list of valid terms with no relationships
* Ring - A list of valid Terms that are all Equivalent
* Taxonomy - A list of valid Terms that are arranged in a Hierarchy
* Thesaurus - A list of valid Terms that arranged in a Hierarchy with optional Equivalence and/or Association between some Terms, usually ordered
* Metadata - Information related to a Term, for example a Period will have a Start Year and End Year

== Translations ==

Translations will be denoted as being in the 'Vocabulary' Domain. The Translations admin will not include these translations, these will be managed in the Vocabulary admin instead.

== Resources ==

* http://www.getty.edu/research/publications/electronic_publications/intro_controlled_vocab/
* https://www.controlledvocabulary.com/

== Database ==

The following tables hold the data for individual vocabularies:
* ark_vocabulary - Key table for vocabulary concepts
* ark_vocabulary_term - Table for terms in a vocabulary
* ark_vocabulary_parameter - Table for optional parameters applying to a term
* ark_vocabulary_related - Table for optional relationships betweens terms if a vocabulary or taxonomy

The following reference tables are unlikely to require editing:
* ark_vocabulary_relation - Reference table for term relation types, i.e.broader/narrower
* ark_vocabulary_type - Reference table for vocabulary types, i.e. list, taxonomy, etc

== Classes ==

*ARK/Vocabulary/Vocabulary - Utility class for static access methods

*ARK/Vocabulary/Concept - Abstract base class for Vocabulary Concept, stored in ark_vocabulary
**ARK/Vocabulary/TermList - Subclass of Vocabulary for list type vocabularies
**ARK/Vocabulary/Ring - Subclass of Vocabulary for ring type vocabularies
**ARK/Vocabulary/Taxonomy - Subclass of Vocabulary for taxonomy type vocabularies
**ARK/Vocabulary/Thesaurus - Subclass of Vocabulary for thesaurus type vocabularies

*ARK/Vocabulary/Term
*ARK/Vocabulary/Parameter
*ARK/Vocabulary/Related

*ARK/Vocabulary/Type
*ARK/Vocabulary/Relation

== Classes ==

ARK2/Avalon2

2016-12-15T21:09:38Z

John Layt: /* Avalon 2 */

= Avalon 2 =

Avalon is the project management ARK for LP Archaeology. Avalon triggers the creation of a job folder on the local server, which is then synced to the other servers. If a GIS is required, then standard shapefiles are created in the job folder, and all the GIS data is stored in the job folder. Any Project that also requires a site recording ARK then sets that up separately.

A number of issues exist here:
* File syncing is slow, resource intensive, and error prone
* GIS data is heavily duplicated, wasting space
* GIS data is fractured, i.e. finding what data we already have requires manual searching of folders, and may lead to the purchase of duplicate data
* GIS Figures are still too complex to produce and could be automated more.
* Use of ARK for data recording is time-consuming to set up so often not done for small jobs.
* Many jobs are deemed too small to bother entering the data in ARK, this makes querying our data corpus or automating many report writing functions impossible
* Versioning files, submitting them for review, and checking they have been is a manual process and open to abuse

ARK 2 provides several new features that can alleviate this:
* File management and file versioning support
* Workflow management
* Easy automated creation of new ARK instances
* Multiple sites and schemas in a single ARK install
* Support for running on PostgreSQL, and consequently support for PostGIS

The proposal for Avalon 2 is as follows:
* A single ARK install running both Avalon and all site ARKs, either as a single instance or two instances sharing a single user database.
* The database will run on PostgreSQL, providing both the Avalon/ARK data server and a PostGIS database in a single install.
* The GIS database will manage as many shared vector and raster resources as possible, as well as the GIS for each Project.
* On creation of a new Project, the PM can choose to create a GIS and an ARK, or trigger this at any time later.
* All the ARK site data will be stored in a single instance using multiple standard company data schemas, both standardising recording methods and enabling search across all projects.
* A connection to Harvest will be created using their API, allowing for combined reporting, and automated creation of projects and allocation of staff.
* Later Avalon could seek to replicate the required features of Harvest itself.

* Integrated Wiki to replace Mediawiki

* https://github.com/charlesroper/OSGB_Grids

ARK2/UX

2016-12-12T09:43:24Z

John Layt: /* Map View */

= UX =

ARK, the Archaeological Recording Kit, is a web application for recording, analysing, and publishing archaeology data. ARK provides a very flexible platform akin to a CMS to allow different clients to configure their website to meet their data recording needs. This flexibility applies to the UX as well as the data schema.

ARK 1 is largely custom code that is showing its age and hard to customise. ARK 2 is a ground-up rewrite to bring ARK into line with modern web technology including a REST API, while also making it easier to configure and maintain. ARK 2 will also enable ARK to be offered as a hosted service.

While clients will be able to build custom web or app interfaces using whatever technology they prefer, most will choose to use and possibly adapt the default Bootstrap/Twig web interface which needs to be both feature-complete and flexible, albeit at the cost of some efficiency. They are also likely to use variations on the standard 'Site Recording' data schema with 'Sites' or 'Projects' at the top level, as will the hosted ARK-as-a-service, so the default UX should be weighted towards that scenario.

Note that 'Site' is a term often used in archaeology to define a geographic area of interest or excavation, to avoid confusion we will try refer to 'An ARK Website' and 'An Archaeological Site' wherever possible.

== Data Schema ==

The ARK data schema is configurable using the concept of Modules. A Module can be thought of as an Object in the Object-Oriented model, a Resource in the REST model, or a table in the Relational model. A Module has a set of data properties and relationships to other Modules. An instance of a Module is known as an Item. The schema is completely configurable by the website admin, but a number of pre-defined configurations for industry-standard recording schemas are available to use.

=== Field Recording ===

The primary use for ARK is recording archaeological data in the field, analysing the data in the office, and publishing the data online. It reflects the paper-based field recording systems originally used in Archaeology, being based around Registers and Forms, and is designed to replace part or all of a paper-based system. Field staff are often freelance or on short-term contracts, so the UX needs to be familiar enough for them to adapt to quickly. The UX needs to balance the familiarity of the users with the paper-based systems with the improved workflows that can come from an online system. The UX should also be adaptable to other field data recording scenarios both in archaeology, and also other potential uses such as conservation.

[[File:ExampleRegisterSmall.png]]
[[File:ExampleSheetSmall.png]]

The main recording system used is called Single Context Recording in which various objects and concepts found on an archaeological site are recorded.

* [https://en.wikipedia.org/wiki/Context_(archaeology) Context] - A basic unit of archaeological recording, such as a layer of soil or a wall.
* Photo - A photo that can be linked to the objects seen in the Photo, such as Context, Timber, Find, etc.
* Plan - A drawn plan of an object, usually a Context or Find, which can be linked to those objects.
* Section - A drawn cross-section of contexts
* Sample - Part of a Context saved for later scientific analysis in the laboratory.
* Find - A human made object that is found in a Context, like a pot or a coin.
* Timber - A piece of timber used in a building or other man-made structure.
* Subgroup - A collection of Contexts that make up a minor feature or object.
* Group - A collection of Subgroups that make up a major feature or object.

This could be drawn as:

[[File:MinoriesSchema.png]]

Other archaeological objects can also be added to the model as required.

As staff in the field excavate a site, they initially Register the basic details of contexts and objects using phones or tablets, and can then add more details as required. An alternative use model is to record on paper and later copy into the ARK. Staff in the office then process that data, such as putting the Contexts into Subgroups and Groups, before publishing the results. The results are mostly used internally for writing formal reports, but may also attract wider professional or public interest.

== User Profiles ==

* Anonymous - A member of the general public interested in browsing information about the archaeological site. Primary requirement is rich detail with easy navigation.
* Field Operative - A member of staff recording and editing data in the field. Primary requirement is speed and simplicity of data entry in challenging conditions (protective gloves, rain, mud, etc), often only using a mobile phone.
* Field Supervisor - A member of staff supervising other field staff with supervisor-level site functions. Primary requirement is quickly and easily reviewing and editing data.
* Office Operative - A member of staff in the office processing data. Primary requirement is full-featured but efficient data entry and editing, but also search and navigation.
* Office Supervisor - A member of staff in the office doing higher level data analysis. Primary requirement is powerful search and navigation functions.
* Website Admin - An administrator of the ARK website. Primary requirement is basic admin functions, mainly user admin and vocabularies, but sometimes also schema.
* System Admin - An administrator of the ARK install (Different from Site Admin on multi-site/multi-tenant installs). God.

== Configurable Views ==

Just as the data schema is completely configurable by a website admin, so too are the views of that data. ARK combines a layout grid with content blocks known as Subforms. Each Subform usually contains a layout of data fields from the schema, but can also contain other elements. Subforms are usually used to group fields that are related, or require similar functionality. Different page views of the same Item can be laid out for different purposes, for example the Field Operatives data entry view may differ from the Office Supervisors data analysis view. Where this differs from standard CRUD forms is in the code functionality that can be embedded in a Subform. A different View of an Item can also offer significantly different functionality for that Item.

Item Collection views, such as Table, List, Thumbnail, etc, can also be configured. In ARK 1 the two main variants are currently Register and Search, but ARK 2 will also define a standard List view.

The core UX requirement is to define the relationship and navigation between these core generic pages, rather than the physical layout of the Item fields and Subforms. The current model is not integrated or consistent and leads to considerable confusion for users.

== Generic Pages ==

=== Website Home ===

Landing page for a Website. If private ARK then usually login page. If public ARK then usually some project brochure-ware and links to browse site if anonymous access enabled.

=== User Home ===

Landing page for user after logging in, usually some kind of dashboard, but may be User Profile page.

=== Collection View ===

* List Items by Module
* View modes: Text, Table, Thumbnails
* Filter Items by properties in view
* Select properties to view
* Paging
* Export
* Integrate Advanced Search? (see below)

=== Item View / Edit ===

* View item details
* Edit Item details
* Subforms (think Drupal Blocks)
* Item navigation

=== Map View ===

Spatial data and GIS is core to archaeology, but often in a way that has more in common with CAD than most websites. While some simpler forms of data are recorded, such as find spots or site addresses, most spatial data is a detailed and precise plan of all the contexts in a site drawn to 25mm accuracy. This data needs to be displayed in a way that can help with data analysis, such as how different objects relate to each other, as well as be attractive to a wider public.

OpenLayers is used to display spatial data, both embedded as a subform in an Item View, or on a separate page showing a Collection of Items. Spatial search and interactive editing are also required, but full-blown spatial analysis is done in a proper GIS package that can make calls to the ARK api to obtain the data.

=== Advanced Search ===

Version of Collection View with more advanced search tools, possibly same page with mode switch or hidden panel?

* Cross-module search
* Set based queries
* By Module / Type
* By Property
* By Actor / Action / Event
* By Period
* By Spatial
* Free Text
* Save Search Definition
* Choose from saved Search definitions

== Site Recording Pages ==

=== Register View ===

Short-form add new Item. List view shows most recently registered Items to reflect the traditional paper register.

=== Site/Project Dashboard ===

Dashboard for a Site/Project

== User Security ==

ARK2 will provide a very flexible User Security system using an RBAC model. Website admins will have a choice of registration and authentication options and the UX/UI should adapt to these as configured. Login should work as either a page or a pop-up.

* Register (by self or admin)
* Login
* Optional Captcha
* Optional OAuth2 (Facebook, Google, etc)
* Request password reset
* Profile
** Personal details
** Contact details
** Security details

== Admin Screens ==

See [[ARK2/Admin|Admin Screens]]. UX required is primarily layout and flow, plus User Admin, Translation and Vocabulary pages.

ARK2/Admin

2016-12-07T10:45:47Z

John Layt: /* Translation */

= Admin =

ARK2 offers a stand-alone Admin web interface, allowing an API-only back-end to be configured and maintained. When running the full ARK2 Web frontend, the Admin interface is accessible to admin-level users, but does not 'leak' into the standard user interface. It will probably be an option on the User dropdown menu, or directly at example.com/admin.

Because ARK2 supports multi-tenant/multi-site modes, the Admin interface has two broad divisions between Website Admin and System Admin, i.e. managing an ARK instance and managing the ARK install. How the UX of multi-site / multi-tenant will work is still to be figured out.

The Admin interface should reflect the theme of the main website (colours, fonts, etc), but not otherwise change in layout or function so admin is a consistent experience across sites and installs. Many of the pages will be standard CRUD list views for which we will need a master template, but some pages will benefit from custom UX.

[https://startbootstrap.com/template-overviews/sb-admin-2/ SBAdmin] and [https://almsaeedstudio.com/ AdminLTE] were early options assessed. The Drupal Admin panel is another example.

== Profiles ==

The options available in the admin frontend will vary depending on the admin level of the user.

* Hosted Website Admin - Basic website details, add users, but not able to change schema, roles, etc.
* Website Admin - Full website admin for a single website, including schema
* Multisite Admin - Full website admin for multiple websites?
* System Admin - Installation admin

== Menu ==

Rough menu or grouping structure, not all features required in 2.0.

Admin
- Website
- Content
- Structure
- Security
- System

Initial view should be some kind of dashboard summary view of the install, alternatively the Website initial page.

== Website ==

Administration of an ARK Website (Instance? Project?).

- Website
-- Details
-- Theme
-- Region
-- Alerts

Initial view should be some kind of dashboard summary view of the websites managed by the user with ability to switch websites, alternatively the Website Details.

=== Details ===

General website details:

* Name
* Short description
* Logo
* Favicon
* Contact Name / Email
* Copyright
* Content license: CC, etc

=== Theme ===

Web frontend details:

* Frontend (API, Admin, Flat, Web) [Not available for Hosted ARK Admin]
* Skin (Colour, etc)

=== Region ===

Regional and language settings:

* Locale
* Time Zone
* Multilingual site
* Default language
* Supported Languages

=== Alerts ===

Alerts or Flashes are notifications displayed to users, i.e. [http://getbootstrap.com/components/#alerts Bootstrap Alerts]. Site or System Admins can schedule alerts to be show at a system, site, role, or user level, and configure them to be dismissible and/or persistent. The Admin frontend will provide a page to administer these Alerts. The Alerts Admin will appear at both Site and System levels with the appropriate options.

== Content Admin ==

- Content
-- Static Page
-- File
-- Image
-- Translation
-- Import
-- Upload

=== Static Page ===

Static Page administration. Ability to add or edit a static html page at a given URL.

=== File ===

File options and file management. Allowed types, sizes, etc. Flysystem config. Basic CRUD list with admin options. Bulk upload option?

=== Image ===

Image options management (files themselves included in File tab). Toolkit options, thumbnail profiles, regenerate thumbnails, etc.

=== Translation ===

ARK2 uses [http://symfony.com/doc/current/translation.html Symfony Translation] for translating both markup and content into multiple languages, including data values such as taxonomy terms. A Site Admin needs to be able to add and maintain translations directly. Advanced features are not required, as specialised websites will be used to maintain the core translation files. A sortable/filterable list view of translation keys and their translations into the site's supported languages is required. Useful features would including listing keywords that are missing translations for a given language, export al translations to XLIFF file, etc.

A translation has the following fields:
* Domain - Message group, e.g. Users, Admin, Errors, etc
* Keyword - The translation key
* Role - The translation role, e.g. Default, Title, Description, Opposite, etc
* Language
* Text - The translation
* Notes - Notes to help the translator, will default to original English translation if not otherwise set

Note that a Keyword can have multiple Roles in the table, for example keyword 'field.location' will have Roles for 'default' (the translation to use for any Role not set), 'title' (the Title to show for a field, in title case), 'description' ( a short description of the field that can be shown in a help mouseover).

Some possible examples:
* https://github.com/lexik/LexikTranslationBundle
* https://github.com/instaclick/TranslationEditorBundle
* https://www.transifex.com/
* https://demo.weblate.org/

=== Import ===

Bulk data import functions.

=== Upload ===

Bulk file upload function, or in File section?

== Structure ==

Section to manage the data structures available, i.e. Modules, Schemas, Fields, etc.

Most pages to be developed for v2.1, but Vocabulary may be required for 2.0.

- Structure
-- Vocabulary
-- Format
-- Schema
-- Workflow
-- View

=== Vocabulary ===

Page to manage controlled vocabularies (List, Ring, Taxonomy, Thesaurus), including translations (i.e current Aliases). See the [[ARK2/Vocabulary|Vocabulary]] page for more details.

=== Format ===

Page to manage data field formats.

=== Schema ===

Page to manage Module Data Schemas.

=== Workflow ===

Page to manage Workflows.

=== View ===

Page to manage Module View Layouts.

== Security Admin ==

Section to manage the user and security.

Required for 2.0.

See the [[ARK2/Security|Security]] page for more details.

- Security
-- Options
-- Users
-- Roles
-- Permissions

ARK2 implements a Role-Based Access Control system, where Users are assigned Roles, and Roles have Permissions. The usual set of functions will be required for both self-registration and invitation-only models, password resets, and user management.

[http://www.userfrosting.com/ User Frosting] was an early option assessed.

Note that a User Profile page will be available in the main Web frontend, the user view in the Admin frontend should only be focussed on the security admin aspects of the user, but should share a design language with the User Profile.

=== Options ===

Configure security options, such as user registration policy, anonymous users, etc.

=== Users ===

List of users and management tools. Register, approve, reset password, suspend, delete, etc.

=== Roles ===

List of roles and management tools.

=== Permissions ===

List of permissions and management tools.

=== User Registration ===

Register a user, used for both Admin-Register or Self-Register.

=== User Settings ===

The user's admin-serviced security settings, such as roles, password reset, etc.

=== User Profile ===

The users profile, including self-service security settings, personal details, avatar, notifications, activity stream, social media accounts, etc. Part of the main user facing UX.

== System Admin ==

Section to configure system-level settings.

Only used by Sysadmin level users.

Not required for 2.0, instead sysadmins will use the Sysadmin Console.

Options in the Sysadmin panel will include:

- System
-- Status
-- Region
-- Alerts
-- Messages
-- Site Defaults
-- Cron
-- Logging
-- Reports
-- Routing

=== Status ===

Install status, maintenance mode / live mode, etc.

=== Region ===

Install-wide default regional and language settings:

* Locale
* Time Zone
* Multilingual site
* Default language
* Supported Languages

=== Alerts ===

Set install wide alerts, i.e. maintenance announcements.

=== Messages ===

Message maintenance, similar to translations.

=== Site Defaults ===

Default settings for new sites when in multi-site mode.

=== Cron ===

Cron job management.

=== Logging ===

Manage and view logs.

=== Reports ===

Various status reports.

=== Routing ===

Configure routes, api, etc.

ARK2/Spatial

2016-11-27T13:52:24Z

John Layt: /* Design */

== Requirements ==

Basic spatial support is required, e.g. storing find points or area bounding boxes, as well as the advanced mapping interface options, such as spatial search and processing. We use the [http://www.opengeospatial.org/standards/sfs OpenGIS standard] as defined by the OGC and implemented by various spatial databases.

Basic back-end support:
* Lossless storage of standard geometries, including CRS
* Validate geometries
* Interchange geometries via API

Advanced back-end support:
* Spatial search
* Spatial processing
* File import/export

== Design ==

The lossless storage and interchange of spatial data is a core function, but cross-database support is poor:
* MySQL/MariaDB has a native GEOMETRY type, but this is only 2D
* PostgreSQL and SQLite have 3D support, but only by adding extensions (PostGIS and Spatialite)
* Adding extensions may be difficult on hosted platforms, or the additional spatial functions provided may not be needed by a site

The lossless storage of geometries in the database therefore needs to use standard datatypes in either WKT or WKB format. The downside to this is that the data cannot then be used for spatial search or processing within the database. If spatial search or processing is required, then the data will need to be duplicated in proper spatial fields.

Again, support for spatial search and processing is uneven:
* MySQL supports standard SQL spatial functions from 5.6 onwards, and spatial indexes for InnoDB from 5.7.6 onwards
* MariaDB supports standard SQL spatial functions from 5.5 onwards, and spatial indexes for InnoDB from 10.2.2 onwards
* Both MySQL and MariaDB support spatial indexes in MyISAM from 5.5 onwards
* PostgreSQL and SQLite support spatial functions but only by adding extensions (PostGIS and Spatialite)

As spatial search and processing are extended functions, enabling them will require installation on an appropriate platform, however fallbacks will be provided to the extent a platform supports them:
* On PostGIS and Spatialite the full native facilities will be used on a copy of the data
* On MySQL between 5.6 and 5.7.6 and on MariaDB, a MyISAM table will be used to store a copy in a GEOMETRY field with a spatial index and processed using the native spatial functions
* On MySQL 5.5, a MyISAM table will also be used, but the functions will either be the limited set available, or using GEOS if available
* On PostgreSQL and SQLite if GEOS is available then it will be used on the original data
* Otherwise spatial search and processing will be unavailable
* An option is to ship Spatialite with ARK and run it as the spatial engine only

A useful side-effect of this strategy is that the spatial search table could be stored on a separate database server, such as a local Spatialite instance or a shared PostGIS server.

Idea to investigate: if using Redis for cache, then has geospatial cache which can be used for spatial search in bounding box or radius.

== Implementation ==

A number of OpenGIS compliant libraries do exist, but none that meet our full requirements:
* https://github.com/brick/geo - OpenGIS geometry library, requires GEOS, MySQL, PostGIS, or Spatialite. Has WKT and WKB support, but no file import/export support. Provides DBAL data types. Current but low-level maintenance, but not widely used.
* https://github.com/phayes/geoPHP - OpenGIS geometry library, internal routines but supports GEOS, lots of basic file import/export 2D only, effectively unmaintained but widely used, would need to fork and do major cleanups.
* https://github.com/creof/doctrine2-spatial - Modern, currently maintained, Doctrine ORM integration, but MySQL and PostGIS only. Could write backends for spatialite and GEOS?
* https://github.com/symm/gisconverter (and several others) - Various file importer libraries with partial OpenGIS class support but no functions.

Likely solution will be to use Brick and fork various file libraries to work with it. May also fork geoPHP to work as a Brick backend? Alternative is major refactor of geoPHP, or piecemeal integration of each converter library.

Front-end support will continue to use OpenLayers.

== Platform Support ==

* MySql >= 5.5 supports 2D geometry & MyISAM spatial index, >= 5.6 spatial processing, >= 5.7.6 for InnoDB spatial indexing
* MariaDB >= 5.5 supports 2D? geometry and spatial processing natively, spatial index in InnoDB >= 10.2.2 https://mariadb.com/kb/en/mysqlmariadb-spatial-support-matrix/
* PostGIS native 3D geometry, spatial index, and spatial processing
* Spatialite native 3D geometry, spatial index, and spatial processing
* Postgresql no support, fake using WKT storage
* SQLite no support, fake using WKT storage
* GEOS provides spatial processing

ARK2/Files

2016-11-16T11:20:51Z

John Layt: /* File Management */

= File Management =

File and Media management have seen major changes in ARK2, with a more flexible system implemented.

* Files are now a core Module instead of a dataclass and look-up table, allowing for flexible metadata based on file type, and using files as either data properties or relationships.
* The file system is abstracted using [http://flysystem.thephpleague.com/ FlySystem] allowing files to be transparently saved locally or to cloud providers.
* File versioning is supported.
* Image manipulation is abstracted using [http://glide.thephpleague.com/ Glide] and [https://github.com/avalanche123/Imagine Imagine] which supports ImageMagick or GD.
* Thumbnails are stored separately from the master files to make backups, archiving, and regeneration easier
* Multiple thumbnail profiles can be configured
* Thumbnails will be generated asynchronously, so can either be queued for batch processing, delayed until a suitable time window, or only generated when needed.

The Media Types and File Types supported are configurable, with the [https://www.iana.org/assignments/media-types/media-types.xhtml IANA standard types] provided by default:
* Image (jpg, png, etc)
* Audio (mp3, etc)
* Video (mkv, etc)
* Text (txt, etc)
* Application (pdf, odt, ods, etc)

In multi-tenant mode, the files for each ARK instance are stored separately under the 'sites' folder, e.g. 'sites/mysite/files'.

The directory structure for each instance is as follows:

- files
|- download
|- <user>
|- upload
|- <user>
|- tmp
|- <user>
|- data
|- <mediatype>
|- <tokens>
|- cache
|- <profile>
|- <mediatype>
|- <tokens>

where the <mediatype> is as defined in the configuration, and the <token> is a configurable token to split the folders into manageable sizes. By default the <token> will be split by file id in groups of 1000, so subfolders for 0, 1000, 2000, 3000, etc.

The individual file names will be in a standard format. Flysystem abstracts folders and file names as a relative path, which will take the form:
files/data/<mediatype>/<token>/<id>.<revision>.<suffix>
files/cache/<profile>/<mediatype>/<token>/<id>.<revision>.<suffix>

There will be no direct file or image links, all files are considered private and requests to the mapped URL will be security checked. The mapped URL forms will be:
files/<id>/<name>
files/<id>/<basename>_<profile>.<suffix>

A custom Flysystem manager is provided that allows different filesystems to be transparently mounted at different points in the file path. For example, the root files/ path is mounted locally, but files/data/ is mounted on Amazon S3.

== Metadata ==

Full metadata is supported in line with Dublin Core.

== Versions ==

== Design Work ==

* Data downloads / exports
* Documentation files
* Temp files with expiry
* Mapping files

Very similar to http://documentation.concrete5.org/developers/working-with-files-and-the-file-manager/overview

ARK2/Branding

2016-11-14T13:41:02Z

John Layt:

= Branding / Community =

Two potential issues suggest an evolution of the ARK brand is required
* Building a development and support community may be easier if ARK is branded as a stand-alone project, rather than seen as owned/controlled by LP Archaeology
* Extending use of ARK and Hosted ARK to areas outside archaeology may be held back by emphasising the archaeology aspect in the branding

Branding as something like 'The ARK Project, sponsored by LP Archaeology', and coining a bacronym like 'ARK Recording Kit' might solve these issues.

The Hosted ARK would need a separate identity to the development project to keep the Open Source / Commercial split clear. A simple .com vs .org difference is probably not clear enough. Examples include:
* Salesforce.com vs Force.com
* Mediawiki vs Wikia
* Wordpress .com vs .org

Branding would thus consist of three parts:
* The project
* The products
* The service

The branding would need to be distinctive but consistent to make it clear they are part of a cohesive whole.

Words with Ark / Arc in them (but not arch) for possible project or theme names:
* Archive / ARKhive
* Arctic (very white/light theme?)
* Arcadia
* [https://en.wikipedia.org/wiki/Arkose Arkose] (type of sediment)
* Arkaeology (available in .com, .org, .net!)
* Arcade
* Archaic / Arcane / Arcanum / Arcana (more apt for ARK v1 ;-)
* Arcuate / Arcuated (Arc/bow shaped)
* Architrave
* Architect
* Archipeligo
* Archosaur
* Arktivity...

Possible ARK Project / Community names:
* ARKadia
* ARKaeology
* ARKforge
* ARKhitect
* ARKhive
* ARKhub
* ARKproject (not available on github)
* ProjectARK
* BuildARK
* Project Indiana
* Hangar 51

ARK2/Design

2016-11-14T13:02:50Z

John Layt: /* Spatial */

= Design =

High level design decisions for ARK2.

== Supported Platforms ==

ARK2 will be built using modern PHP standards to support multiple Operating Systems and Database Systems. ARK will only actively support platforms that are actively supported by their maintainers. ARK may work on earlier versions but this is not guaranteed.

For more details see the [[ARK2/Technical|Technical Details]] page.

== Framework ==

ARK2 implements a new RESTful Request/Route/Response skeleton using a Front Controller model and token-based security, based on an external micro-framework and components adhering to the PSR standards and managed via Composer. This will reduce the amount of code maintained internally, update the code-base to modern web-app design principals, and provide a degree of future-proofing by allowing switching of components.

The [http://silex.sensiolabs.org/ Silex] micro-framework and [http://symfony.com/ Symphony] components have been selected due to their high quality, long support history, and wide use and support

For more details see the [[ARK2/Technical|Technical Details]] page.

== Database Abstraction ==

ARK2 will support the use of MySQL, PostgreSQL or SQLite as the database by using the [http://doctrine-orm.readthedocs.io/ Doctrine ORM and DBAL] database abstraction layer.

More details can be found on the [[ARK2/Database]] page.

== Security ==

ARK2 will require a new security solution, This solution will be token based, support username/password and OAuth2 authentication, and use Role-Based Access Control for authorization. HTTPS will be required using Let's Encrypt as the CA..

For more details, see the [[ARK2/Security|Security]] page.

== REST / HATEOAS / Hypermedia ==

An evolution of the ARK data model and API to try realise the [http://ark.lparchaeology.com/hypertext/ full ARK vision] will be based arround the Hypermedia concepts of [https://en.wikipedia.org/wiki/Representational_state_transfer REST] and [https://en.wikipedia.org/wiki/HATEOAS HATEOAS] as developed by Roy Fielding in his doctoral thesis in 2000. These concepts include resources, relationships, state, and discoverability, and are closely related to the Semantic Web. ARK modules will evolve to represent Resources that can be linked in together in the current flat relationship structure, or organised into configurable hierarchies such as the default Site/Module/Item mostly used by ARK instances. These concepts will be most easily exposed through a RESTful API.

For more details see the [[ARK2/API|API]] page.

== Frontend ==

The frontend will be migrated to [http://getbootstrap.com/ Bootstrap], [https://jquery.com/ jQuery], and [http://twig.sensiolabs.org/ Twig], the most popular and well-supported frontend ui component and template systems. This will allow for easier customisation of ARK's appearance by third parties.

For more details, see the [[ARK2/Frontend|Frontend]] page.

== Migration ==

A migration process from ARK 1 to ARK 2 will be provided.

Data migration. Existing tables will need to change from MyISAM to InnoDB. Change in place carries a degree of risk of data loss if the migration fails part way. Attempting to restart failed migrations is also prone to error. To protect users data, a new database will be created with new tables and the data copied across. Should migration fail users will easily be able to roll back to their old install, or keep retrying the migration until it does succeed. In effect the ARK init script will be run, followed by the migration script.

User migration. Users will be migrated from LiveUser to the new RBAC system. This will require a compatible default user config.

Config migration. A config migration script will be provided, but may require adapting for individual ARKs.

== Sysadmin Console ==

An sysadmin console will be provided for use on the command line. This will provide a number of tools:
* Database administration, such as creation, migration, backup, etc
* MultiArk installs
* ARK wide alerts
* Maintenance mode (immediate and schedule)
* Upgrade tools
* etc

Equivalents for some of these functions will be provided in an ARK Sysadmin panel (separate to the ARK Admin panel):
* ARK wide alerts
* Maintenance mode (immediate and schedule)
* etc

== Translations / Localisation ==

A key to providing Ark-As-A-Service will be translating the user interface and schemas into as many languages as possible to maximise the potential user base. ARK1 does not have the tools to make this process easy to perform or manage, and it would be a waste of resources to build them. It is recommended to use one of the existing online open-source translation projects to crowd-source the translations. This will allow interested parties and potential clients to translate ARK for themselves and to grow a local community to support ARK in their country.

Changes will be made to the translation process to bring ARK into line with industry best practices and tooling, allowing for common features such as correct plural forms. This will be based on the Symfony Translation componant.

More details can be found on the [[ARK2/Localization]] page.

== Action Logging / Activity Streams / Gamification ==

All user actions and events in ARK will be logged, enabling an standard Activity Stream at user level that can support gamification. The actual gamification is considered outside of the scope of core ARK2. Some inspiration for the implementation can be taken from Event Sourcing, but a full implementation will not be attempted.

* http://martinfowler.com/eaaDev/EventSourcing.html
* http://activitystrea.ms/
* https://www.w3.org/TR/activitystreams-core/
* https://github.com/barnabywalters/php-activitystreams
* https://github.com/redpanda/ActivityStreams
* http://jwage.com/post/54480997504/building-activity-streams
* https://dev.playlyfe.com/guides/getting-started.html

== Command Bus Architecture / Application Services ==

A Command Bus / Event Bus architecture will be implemented to support execution or queuing of synchronous and asynchronous Commands and Events. Console Commands will be wrappers around Commands that parse input from the command line.

* http://php-and-symfony.matthiasnoback.nl/2015/01/a-wave-of-command-buses/
* https://github.com/SimpleBus/
* http://tactician.thephpleague.com/
* http://culttt.com/2014/11/10/creating-using-command-bus/

== Workflow Management ==

Stretch goal. Needed for Avalon. Packages exist to define workflows using state machines.

Possible workflow scenarios:
* Post-ex
* Sample taking
* Avalon jobs and documents
* Data checking

== Document Management ==

Stretch goal. Needed for Avalon. Management of documents and versions a la Sharepoint. Try use CMIS standard as used in LibreOffice app, or WOPI used by LibreOffice online.

* https://www.alfresco.com/cmis
* https://packagist.org/packages/dkd/php-cmis
* http://wopi.readthedocs.io/projects/wopirest/en/latest/
* https://kolabian.wordpress.com/2016/11/16/collaborative-editing-with-collabora-online-in-kolab/
* https://www.collaboraoffice.com/code/
* WebODF?

== CMS / Blog / Project Websites ==

Either provide an OoB Wordpress integration and host client websites, or add features to ARK to provide the basic project website features (home page, blog, contact us, calendar).

* https://github.com/bolt/bolt CMS built using Silex/Symfony

== Errors ==

Standardised error codes will be used, based on the JSON API standard format. Error codes will be stored in a database table and exported via the API, with actual error messages translated via the standard methods, with detailed debug/help available on the ARK wiki via standardised links. Fatal errors will be thrown as exceptions, with the Controller responsible for catching the error and reporting it in the appropriate manner, i.e. via web or api. Non-fatal errors such as validation errors can be batched before return. While numeric error codes are convenient, they make code hard to read and debugging tracebacks harder, so all error conditions should be explicitly commented in the code, and error codes should be as unique as possible..

== Matrix / Graph ==

PHP:
* https://github.com/clue/graph
* https://github.com/graphp/algorithms
* https://github.com/cpettitt/graphlib/wiki/API-Reference

== Spatial ==

Basic spatial support is required, e.g. storing find points or area bounding boxes, as well as the advanced mapping interface options, such as displaying spatial layers, spatial search, etc. See [[ARK2/Spatial|the Spatial page]] for more details.

ARK2/Model

2016-10-30T15:14:56Z

John Layt: Replaced content with "= Model ="

= Model =

ARK2/Architecture

2016-10-30T14:57:47Z

John Layt: /* Architecture */

= Architecture =

ARK2 has been built to modern web standards using the latest Object-Oriented PHP development methodologies. This includes using widely supported frameworks and components in a Front-controller architecture.

== Frameworks and Components ==

ARK2 is built on the Symfony components using the Silex Micro-Framework to manage the services and routing.

=== Front-Controller ===

Single page, mod-rewrite, run application,builds and renders requested page.

=== Container ===

Why container.

Using http://pimple.sensiolabs.org/

Dependency Injection, etc http://php-di.org/doc/understanding-di.html

=== Router ===

Maps url requested to list of url patterns, then dispatches request to Controller that can fulfil that request.

=== Controllers ===

=== Model ===

=== View ===

=== Object-Relational Mapping (ORM) ===

=== Database Abstraction ===

== Multi-Tenancy / Multi-Site / Multi-Config ==

A number of architectural issues surround Multi-Tenancy, Multi-Site and Multi-Config in an ARK instance. These primarily affect how a hosted ARK service will be run, but also how a standalone organisation will manage their ARK instances.

* An ARK instance is here defined as a combination of ARK users and the ARK site data they are able to access, usually under a single project/brand/organisation.
* A database is defined as a combination of a database user and the tables it can access, not the database server instance which can hold multiple database.
* Multi-tenancy is the ability to have multiple ARK instances in a single ARK install.
* Multi-site is the ability to have multiple sites within an ARK instance.
* Mulit-config is the ability to have multiple ARK schemas within an ARK instance, i.e. different sites having a different config.

Choosing an architecture involves a series of trade-offs around ease-of-development versus ease-of-maintenance. The simplest solution is the current structure, where an ARK instance has a single tenant with a single config across multiple sites. There are problems with this however:
* Each instance requires a separate code install, database and URL
* If a single organisation wants multiple ARK schemas (say trench-based rural and a full urban SCR) they must run separate ARK instances for each schema, meaning users must remember which instance has which sites and maintain separate user IDs, and the apps using the API must know this as well.
* Making significant upgrades to an organisation's config requires a separate ARK install
* Scaling up to 100's of instances creates 100's of installs and 100's of databases which will make support difficult and expensive even with automation

At the opposite extreme is an architecture where a single ARK install supports multiple tenants, sites and configs in a single database. While this solves the above issues by greatly simplifying maintenance there are a number of issues here too:
* Code and SQL is significantly more complicated, joins especially become difficult
* Key bloat on all tables as fields required for tenant and site which may affect performance
* Table bloat with all data being in a single set of tables which may affect performance
* Back up and archive is an issue as the data for different tenants needs to be separated, probably requiring custom code instead of standard tools
* Security is an issue with data access control now occurring in the app code
* A single tenant can overload the server and take all tenants down
* Distributing load across servers becomes difficult if not impossible
* Upgrading an install means all site configs must be upgraded too, you cannot leave a site on an old version
* Existing code and data would make ARK1 migration far more complex

A half-way house model would be to allow a single install to have multiple tenants, but each tenant has its own database instance:
* A simple key structure is kept, keeping the code simple
* Each tenants data is kept separate, solving the size, security and backup issues
* Load can be easily distributed by moving a tenant to another server by simply moving their database and/or redirecting their url
* Code maintenance is kept simple, but database management becomes more complex again
* Upgrading an install will still require upgrading all sites

Note: A practical limitation is imposed by MySQL and SQLite support which only allow a single 'namespace' per database, unlike PostgreSQL and others which support multiple 'namespaces' which would allow each tenant to have separate sets of tables within the same database.

The strongest case can be made for supporting Multi-Config, primarily as a a means of allowing larger clients to host all their data inside a single install with a single set of users (including LP ourselves). This has several implications however:
* It raises Site Code from an attribute of an item in a module, to being a key at a higher level than the modules themselves, i.e the modules available will change depending on the Site Code
* As a consequence it substantially changes the api to add the site code above the module
* It may make searching across site codes difficult
* ???

The full combination would allow a hosted ARK solution as follows:
* Lowest price tier (£5) / mass market / community dig type sites are hosted in a single multi-tenant install, only allowed a single site/config, may not allow own domain?
* Upgrade to lowest tier (£10) still in single multi-tenet install, but allowed say 5 sites/configs, maybe allow own domain?
* Next tier(s) (£15/£20/£25?) gives separate install, probably in own virtual host, own domain, with unlimited sites/configs?
* Possible top-tier for large-scale sites with guaranteed support contract

This would keep the maintenance burden on the lowest-profit sites to a minimum, while encouraging up-sells as and when needed.

Install management could be simplified by developing a set of built-in tools.
* Installs using git, run git pull to upgrade
* Doctrine migrations enable auto data updates
* Auto-check function for new releases and notify admin
* Admin panel to put site into maintenance mode, run code update, run data update

Splitting database roles may assist in this:
* User database - allows Multi-tenant to choose if shared users for all/some tenants, or any tenant to have own users
* Config database - The ARK configuration, schemas, forms, etc, allows multi-tenant to share all configs with all/some tenants, or any tenant to have own set
* Data database - The ARK data, each tenant will have their own database

The framework will manage three separate database connection variables, but where the database roles are shared by a database instance then the connection objects will be the same.

ARK2/Frontend

2016-10-30T14:40:00Z

John Layt: /* Scripts */

== Overview ==

ARK2 has imposed a clear separation between the backend and frontend, allowing the frontend to operate completely independently. ARK2 provides a number of standard frontend implementations, but sites can adapt or implement their own. This also supports multi-tenant mode where a single ARK2 install can be configured to run multiple ARK2 sites using different frontends.

Two options exist for building frontends:
* Integrated frontends using the ARK2 PHP API via TWIG templates and Views defined in the config database
* Standalone frontends using calls to the ARK2 REST API

The integrated frontends that ship with ARK2 are implemented in [http://getbootstrap.com/ Bootstrap], [https://jquery.com/ jQuery], and [http://twig.sensiolabs.org/ Twig], the most popular and well-supported frontend libraries. Frontends can be cloned and modified for a specific site, allowing for easier customisation of ARK's appearance by third party designers. Standalone frontends could be implemented in any other technologies such as React and Vue.

The integrated frontends shipped with ARK2 are:
* admin - A minimal Web Admin front-end for configuring an API-only server
* ark2 - The default front-end supporting the full ARK2 feature set
* bootstrap3 - A minimal Bootstrap 3 frontend designed as a base for custom frontends
* bootstrap4 - A minimal Bootstrap 4 frontend designed as a base for custom frontends
* flat - A simple, non-js, read-only front-end designed for scraping, e.g. for static archiving or web-bots

The integrated frontends require a build process to assemble the required frontend resources into a distributable form. This page documents the process required to do so.

The resources built into a frontend can be obtained from three sources:
* Vendor resources: Third-party libraries from outside ARK2, such as Bootstrap, jQuery, and FontAwesome
* Core resources: Standard resources provided by ARK2 for interacting with ARK2 features, e.g. common widgets, form layouts, etc
* Custom resources: Custom resources developed for a particular frontend

The build process uses tooling to combine these resources into a single source package that is able to be distributed.

== Source ==

Packaged frontend resources are saved in the src directory by Namespace and Name:

src/<namespace>/frontend/<frontend>

For example, the main ark2 frontend in the ARK name space is stored in:

src/ARK/frontend/ark2

Within each frontend source folder, the source is organised by general type:

|- <frontend>/
|- assets/
|- fonts/
|- images/
|- scripts/
|- styles/
|- bin/
|- console
|- config/
|- credentials.json
|- database.json
|- site.json
|- public/
|- .htaccess
|- index.php
|- templates/
|- translations/

The following source types generally use the default source and remain unchanged:
* bin - Any executables used by the frontend, usually just the site console
* config - The default config files for the frontend, the main site.json file, the database.json file, and the credentials.json file
* public - The default contents of the public webroot folder, including the index.php file

The following source types are usually modified for a custom frontend and rebuilt using the build tooling when changed:
* assets - The public assets for the frontend that get included under the public webroot, i.e. fonts, images, scripts and stylesheets
* templates - The twig templates used to render the html
* translations - Vendor provided translation files

Customising the scripts and stylesheets requires various tooling which is covered in the next section.

== Build Tooling ==

Build tooling for frontend resources is required for a number of reasons:
* Bootstrap can be customised most easily by changing variables used in the Sass templates, which then requires a build step to compile them into CSS
* Production deployment is more efficient if CSS and JS is stripped, merged and minified, while development is easier if a map is generated for the original code
* Resources from multiple sources need to be combined into a single package
* Multiple versions and combinations of resources may be required for performance reasons
* NPM library management downloads the entire package, not just the resources required, an extra step is required to copy just the required resources into the source folder
* All the steps required for packaging and release management can be automated, e.g. clean, compile, tag, package, etc
* The build tooling for the default ARK bootstrap and twig theme can be generalised to allow clients to build and deploy their own customised themes with minimal effort

The build tooling works as follows:
* All build tooling and resources are isolated in the /build folder so it can be excluded from any release packages or production deployments
* Nothing in the /build folder may be depended on by any code outside the /build folder or required for running ARK itself
* [https://nodejs.org/en/ Node], [https://www.npmjs.com/ npm], [http://gulpjs.com/ Gulp] and Symfony Console are used to run the tooling
* NPM is used to manage the Node packages used which are installed into /build/node_packages
* Node packages are used both for the build tools themselves and for vendor libraries providing resources
* Gulp is used to ensure the build scripts run cross-platform
* Symfony Console is used to manage and run the build tasks
* Running tasks will only work inside the /build folder, trying to run outside the build folder will fail
* The resources required to be built for a frontend are defined in a manifest file
* The built frontend resources are then installed into the frontend source directory
* Later, each configured site in the ARK2 install links to the frontend it wishes to use

== Build ==

The front end build is managed by the [[ARK2/Console|Build Console]]. The Build console is in the 'build/' folder and is used to build/rebuild/install a front-end if required. To run the console you must be in the build folder:

cd build
./build

Running the console without a chosen command will show the list of available commands:

<pre>
ARK Build Console 1.9.80

Usage:
command [options] [arguments]

Options:
-h, --help Display this help message
-q, --quiet Do not output any message
-V, --version Display this application version
--ansi Force ANSI output
--no-ansi Disable ANSI output
-n, --no-interaction Do not ask any interactive question
-v|vv|vvv, --verbose Increase the verbosity of messages: 1 for normal output, 2 for more verbose output and 3 for debug

Available commands:
help Displays help for a command
list Lists commands
cache
cache:clear Clear the system cache
database
database:reverse Reverse engineer an existing database as DoctrineXML
frontend
frontend:all Build an ARK frontend. (Args: frontend)
frontend:assets Build the frontend assets (Fonts, Images, Scripts, Styles). (Args: frontend)
frontend:base Build the frontend base (Bin, Config, Templates, Translations, Web). (Args: frontend)
frontend:create Create a new ARK frontend (Args: namespace, frontend).
frontend:scripts Build the frontend scripts (JavaScript). (Args: frontend)
frontend:styles Build the frontend styles (CSS and SASS). (Args: frontend)
frontend:templates Build the frontend templates (Twig). (Args: frontend)
</pre>

The first time you need to build/rebuild a frontend, you need to install the required Node packages. You will need to have installed Node and NPM on your system first.

npm install

To update an existing installation, especially after an ARK upgrade:

npm update

To check the status of your Node environment (including available updates), run:

npm doctor

To install a new node package to use in the frontend, run:

npm install <package>

To build a frontend run the frontend command with the name of the frontend:

./build frontend:all <frontend>

This will install the built front end into 'src/<namespace>/frontend/<frontend>'.

You can also choose to build just one type of resource to save time, e.g. if you only changed a template and don't want to wait for the js to be rebuilt as well:

./build frontend:styles <frontend>
./build frontend:scripts <frontend>
./build frontend:templates <frontend>
./build frontend:assets <frontend>
./build frontend:base <frontend>

The /build folder is organised by resource source as follows:

|- build/
|- core/
|- frontends/
|- <frontend1>/
|- <frontend2>/
|- <frontend3>/
|- node_modules/
|- .jsbeautifyrc
|- .jshintrc
|- build
|- gulpfile.js
|- packages.json

* The /node_modules folder holds the vendor resources managed by NPM as defined in the packages.json file
* The/ core folder holds the core resources provided by ark2
* The /frontends folder holds the custom resources for the various integrated frontends
* The /gulpfile.js defines the Gulp build scripts
* The /build console runs the build scripts
* The /.jsbeautifyrc and /.jshintrc files define the JavaScript coding standards used in ARK2

Within the /core folder and each custom frontend folder, the resources provided are organised in folders by resource source type:

|- <frontend>/
|- bin/
|- config/
|- fonts/
|- images/
|- js/
|- public/
|- scss/
|- twig/
|- xliff/

== Manifest ==

The combination of resources that are built into a frontend are defined by the build manifest.json file in the root of each frontend folder, e.g. /build/frontends/ark2/manifest.json.

{
"frontend": "ark2",
"namespace": "ARK",
"options": {},
"assets": {
"fonts": {},
"images": {},
"scripts": {},
"styles": {}
},
"bin": {},
"config": {},
"templates": {},
"translations": {},
"public": {}
}

Here you see the root level of the manifest defines the same resource classes as is used to organise the frontend source folder, i.e. these sections define the resources that will be packaged together to be installed into the source folder.

There are also three extra root level keys:
* frontend - the name of the frontend used in the source path
* namespace - the namespace of the frontend used in the source path
* options - various options to be used in the build tooling, keyed by the Gulp module that uses them

The default build options are as follows:

"options": {
"uglify": {
"compress": false
},
"sass": {
"outputStyle": "compressed",
"precision": 8,
"includePaths": [
"node_modules/bootstrap-sass/assets/stylesheets"
]
},
"sourcemaps": {
"loadMaps": true
},
"autoprefixer": {
"browsers": [
"Android 2.3",
"Android >= 4",
"Chrome >= 20",
"Firefox >= 24",
"Explorer >= 8",
"iOS >= 6",
"Opera >= 12",
"Safari >= 6"
]
}
},

Not that the autoprefixer settings are those required by Bootstrap and should not be altered.

Within each resource class, you must define what resources are to be included from the three resource sources: vendor, core and custom.

For most resource classes this defines what files to copy into the source folder, either by individual file names or (more usually for core and custom) all files in a folder:

"templates": {
"vendor": [
"symfony-collection/jquery.collection.html.twig"
],
"core": [
"twig/**/*",
"twig/**/.*"
],
"custom": [
"twig/**/*",
"twig/**/.*"
]
},

When copying files in this way, the order used is vendor, core, then custom, so files in core will override files in vendor and files in custom will override files in core. The vendor file paths are relative to /build/node_modules, the core files are paths relative to /build/core, and the custom files are paths relative to /build/frontends/<frontend>.

The scripts and styles groups work differently however as they must call the SASS and JavaScript build tools to compile the destination source code.

The scripts class defines a set of build targets, that is named minified javascript files to be compiled, and the javascript files that are merged to create that file. For example, to build script files ark-default.min.js and ark-map.min.js you could define the following:

"scripts": {
"ark-default": {
"vendor": [
"jquery/dist/jquery.js",
"bootstrap-sass/assets/javascripts/bootstrap.js"
],
"core": [
"js/core-ready.js"
],
"custom": [
"js/frontend-ready.js",
]
},
"ark-map": {
"vendor": [
"proj4/dist/proj4.js",
"openlayers/dist/ol.js"
],
"core": [],
"custom": [
"js/map-ready.js"
]
}
},

The javascript files are merged in the order defined in the groups, so it is recommended to explicitly list the files so that the correct order of dependencies can be defined. Sourcemap files will be created if specified in the options.

The scripts class defines a set of build targets, that is named minified CSS stylesheet files to be compiled, and the list of Gulp build streams required to built by the SASS compiler before being combined into that target file. For example, to build style files ark-default.min.css and ark-map.min.css you could define the following:

"styles": {
"ark-default": [{
"stream": "bootstrap",
"vendor": [],
"core": [],
"custom": [
"scss/vendor.scss"
]
},
{
"stream": "main",
"vendor": [
"font-awesome/css/font-awesome.css"
],
"core": [],
"custom": [
"scss/default.scss"
]
}
],
"ark-map": [{
"stream": "main",
"vendor": [
"openlayers/dist/ol.css"
],
"core": [],
"custom": [
"scss/map.scss"
]
}]
}

The defining of streams within each target allows for the mixing of SASS and CSS source in a defined order so the cascading of stylesheets is applied in the correct order

== Templates ==

Twig template resources are organised using a convention to make dynamic mixing of templates to create many different Views possible.

The twig folder for a frontend is organised as follows:

|- twig/
|- blocks/
|- errors/
|- framework/
|- layouts/
|- pages/
|- profiler/
|- user/
|- emails/
|- layouts/

* blocks - Small chunks of twig code designed to be reused and repeated using Twig Block inheritance, e.g. form fields, widgets, etc.
* layouts - Arrangements of blocks into generic or task specific layouts, e.g. grids, tables, custom content views, etc.
* pages - Definitions of the sets of standard blocks to use when building the HTML document for a particular page
* framework - Definition of the standard HTML document and default page blocks
* errors - Standard error pages, i.e. 404, 500, etc
* user - Required layouts for the user security system, e.g. user verification emails, login forms, etc.
* profiler - Custom templates to be used in the Symfony profiler

The blocks used fall into three general groups:
* Standard page blocks - blocks that define or override standard page elements such as scripts, navbars, etc, which are saved in separate block files.
* Form blocks - blocks that define standard Symfony Form elements such as Textarea, Select, Radio, etc, which are saved in a single blocks/forms.html.twig file
* Custom widget blocks - blocks that define custom widgets that do not use Symfony Forms and are saved in separate block files

The standard page blocks are:
* head - defines the document <head> element
* styles - stylesheets to be included at the top of the document <head> element
* body - defines the document <body> element, usually how the navbar, sidebar, content and footer blocks are arranged in the body
* navbar - defines the navbar or header at the top of the document body
* sidebar - defines the sidebar or menu at the side of the document body
* content - defines the main content of the document body
* footer - defines the footer at the bottom of the document body
* scripts - scripts to be included at the bottom of the document <body> element
* link - defines the link element used in the navbar/sidebar/footer
* flash - defines the page flash element

The standard ARK2 page view is built in Twig as follows:
* framework/doc.html.twig - Skeleton of the HTML5 document, includes undefined blocks for styles, head, body and scripts
* framework/page.html.twig - Extends doc.html.twig and defines the full default set of blocks to use for styles, head, body, and scripts, includes other default blocks used by those blocks such as navbar, but excludes the content block to be used
* pages/<page>.html.twig - Extends framework/page.html.twig to set the content block to be used by a page, and overrides any standard blocks with custom blocks to be used, e.g. override the standard styles or navbar blocks on this page. This is the root template rendered by a page view.

== Scripts ==

The use of Javascript in ARK2 is not very well defined or organised as yet beyond the use of jQuery and various Bootstrap-based widgets.

Core Javascript resources in /build/core/js:
* alert.js - Utilies for working with the ARK2/Bootstrap alerts
* core-ready.js - Initialise the core javascript on document ready
* form.js - Various form related tools, such as the AJAX form mappper
* form-ready.js - Initialise the form javascript on document ready
* location.js - An OpenLayers map picker widget as configured in the ark_map table.
* map-ol4.js - An OpenLayers map widget as configured in the ark_map table.
* map-ready.js - Initialise the map widget on document ready
* pageedit.js - Static page editing using Summernote Editor
* routing.js - The Routing module to use to generate paths from Route IDs.
* utilities.js - Some utility routines, such as debounce.

Common vendor libraries used via NPM:
* jquery
* jquery-form
* moment
* bazinga-translator
* select2
* bootstrap
* bootbox
* bootstrap-datetime-picker
* bootstrap-fileinput
* bootstrap-table

== Styles ==

Styles use a combination of SASS and CSS as required with Bootstrap as the chosen ui style.

== Useful References ==

* https://symfony.com/doc/3.4/form/form_customization.html
* https://symfony.com/doc/3.4/reference/forms/twig_reference.html
* http://www.helloerik.com/the-subtle-magic-behind-why-the-bootstrap-3-grid-works

ARK2/Security

2016-10-30T14:25:14Z

John Layt: /* User Levels */

= Security =

ARK2 uses the Symfony Security component. ARK2 extends this component with its own User Provider and Role-Based Access Control to manage user rights within an ARK.

* User Authentication
** Token-based
** Local user database for stand-alone/internal use
** Distributed Via OAuth and OpenID authentication services (Google, Facebook, etc)

* User Authorisation
** Role-Based Access Control (RBAC) model based on Users/Roles/Permissions

HTTPS is required and is supported using LetsEncrypt to obtain SSL certificates.

=== User Status ===

The User account has the following possible statuses:

* Registered - The user has registered with the website but not yet verified their email (if required)
* Verified - The user has responded to the email verification but not yet been enabled by the admin (if required)
* Enabled - The user is enabled to use the website
* Disabled - The user has been disabled by the admin and cannot login or reset their account
* Locked - The user has failed the password check and must reset their password, or the admin has required them to reset their password
* Expired - The expiry date for the account has passed and must be extended by the admin

=== User Levels ===

ARK User Levels are a renaming of Symfony Security module Roles to prevent confusion with ARK RBAC Roles:

* ROLE_SYSADMIN - System Admin - Admin rights for an ARK system install.
* ROLE_ADMIN - ARK Admin - Admin rights for an ARK instance.
* ROLE_USER - General user rights for any other role within an ARK instance.
* ROLE_ANON - Anonymous users.

== RBAC Model ==

* Permission - A specific right that may be granted to view data or perform an action within the system
* Actor - A persistent entity who can view data or perform actions within the system
* Role - A set of Permissions that can be assigned to an Actor
* User - A security account for logging into the system that can be linked to one or more Actors

ARK2/API

2016-10-29T19:30:32Z

John Layt: /* API */

= API =

ARK2 provides a new, standards-compliant, secure, complete API for query and update.

A number of popular API standards exits. ARK2 has chosen JSON:API as the default API standard, as this is the best suited to the full update functionality required by external apps. However, given the very large user base for JSON-LD within the heritage community, and Google's use of it for SEO, support will be added for a read-only API and semantically-tagged HTML. The library used for JSON-LD will also provide HAL, JSON, and CSV standards support. Facebook's [http://graphql.org/ GraphQL] will be considered in the future.

== Background ==

An evolution of the ARK data model and API to try realise the [http://ark.lparchaeology.com/hypertext/ full ARK vision] will be based arround the Hypermedia concepts of [https://en.wikipedia.org/wiki/Representational_state_transfer REST] and [https://en.wikipedia.org/wiki/HATEOAS HATEOAS] as developed by Roy Fielding in his doctoral thesis in 2000. These concepts include resources, relationships, state, and discoverability, and are closely related to the Semantic Web. ARK modules will evolve to represent Resources that can be linked in together in the current flat relationship structure, or organised into configurable hierarchies such as the default Site/Module/Item mostly used by ARK instances. These concepts will be most easily exposed through a RESTful API.

A RESTful API will be implemented using best practices which are outlined in the following articles:
* http://www.vinaysahni.com/best-practices-for-a-pragmatic-restful-api
* http://blogs.mulesoft.com/dev/api-best-practices-series-intro/

In particular, the following rules will be applied:
* Full level 4 [http://timelessrepo.com/haters-gonna-hateoas HATEOAS REST implementation]
* JSON will be the only format supported
* The [http://jsonapi.org/ JSON API] standard will be used to construct the response body, with [http://json-schema.org/ JSON Schema] standard defining the structure of the data payload.
* API versioning will be used to version the resource path structure, error messaging, and other API infrastructure. The actual data formats will be controlled by the JSON schema which will be available via a standard end-point.
* Authenticated access will only be available using HTTPS, API tokens, and OAuth2
* Read-only unauthenticated unencrypted access will be supported only if explicitly enabled
* Translation keys will be used, with the client downloading a translation catalogue for their required language

The general API URL structure will be as follows:
* /api/v2/<module>/<item>/<submodule>/<item>

e.g. for Site MNO12 and Context MNO12_1 the resource will be www.lparchaeology.com/ark/api/v2/sites/MNO12/contexts/1

A question remains over the structure for hosted ARK installs:
* www.arkhive.org/api/v2/arks/lamas/sites/ABC12/cxt/1 - Allows easy exposure of all instances for LD consumption
* lamas.arkhive.org/api/v2/sites/MNO12/cxt/1 - Easier for migration to stand-alone ARK, more 'private'
* www.arkhive.org/lamas/api/v2/sites/MNO12/cxt/1 - As for 2.

The following HTTP actions will be supported:
* GET - fetch resource
* POST - insert new resource with next available id, i.e. insert a new item with next item number
* PUT - insert or update resource with a specified id, i.e. insert a new item or update an existing item with a set item number
* PATCH - update part of a resource, i.e. update a single field or group
* DELETE - delete a resource
* OPTIONS - What HTTP verbs the current authenticated API user can perform on a resource

The following other endpoints will be supported:
* /filters - The global saved filters
* /filters/123 - The filter definition
* /filters/123/items - The filter result set
* /users - The users
* /users/jlayt - The user details
* /users/jlayt/filters - returns the the list of user filters, etc as per filters endpoint
* /actors - The global actors
* /actors/123 - The actor details

The JSON:API query values will be supported:
* ?filter - basic search in fields (use /filters for advanced search)
* ?sort - sorts results by fields
* ?fields - return selected fields
* ?page - pagination of results
* ?q - not standard, free-text search?

Notes:
* Updating a resource will require some kind of timestamp or last update key to prevent overwriting subsequent changes
* All security / OPTIONS / anon access will be controlled by user roles

JSO Schema Implementations:
* JSON Schema validation via PHP League (simple, complete, extendable) or Justin Rainbow (most widely used)

JSON-API Implementations:
* http://fractal.thephpleague.com/ PHP League Fractal library to serialize as JSON-API - nice, multi-format, but missing some needed features
* https://github.com/woohoolabs/yin - Very nice but PSR-7 and JSON Schema validation uses Rainbow, 5 contrib, 29 releases, last update 2 weeks
* https://github.com/tobscure/json-api - Good, clean, complete standard, agnostic, but no extras/schema/header handling. 17 contrib, 5 releases, last update 1 month, 2nd popularity rank
* https://github.com/neomerx/json-api - 7 contribs, 42 releases, last update 1 week
* https://github.com/nilportugues/php-json-api - 8 contribs, 63 releases, last update 3 weeks
* https://github.com/nilportugues/symfony-jsonapi - Symfony 2 bundle, last update 2 weeks
* https://github.com/mauro-moreno/silex-jsonapi - Silex provider for nilportugues, ast update 2 months
* https://github.com/lode/jsonapi - Simple, 2 contrib, 10 releasses, last update 3 weeks
* PHP Client: https://github.com/Art4/json-api-client

Not considered:
* https://github.com/matthiasbayer/json-api - 1 contrib, last update 1 year
* https://github.com/matthiasbayer/BayerJsonApiBundle - Symfony2 bundle,1 contrib, last update 1 year
* https://github.com/rstgroup/json-api-php - 3 contrib, last update 2 years
* https://github.com/GoIntegro/hateoas deep symfony integration, would be ideal but is driven by Doctrine ORM map and RAML

== Query Parameters ==

The JSON:API standard does not specify how to structure filter/search query parameters, other than to define 'filter=' for this purpose. The following standard, based mainly on [https://developers.google.com/analytics/devguides/reporting/core/v3/reference#filters Google APIs], is used.

Operators (must be encoded where appropriate):
== Equals/ Exact match
!= Does not equal / Does not match
> Greater than
< Less than
>= Greater than or equal to
<= Less than or equal to
=@ Contains substring
!@ Does not contain substring
=~ Contains a match for the regular expression
!~ Does not match regular expression
! Not
, Or
; And
() Grouping
[] In
'' Literal string

For example:

&filter=foo==3,foo==5;bar=@bong;magic[this,that,other]

Calling functions, such as beginsWith() or intersects() could be added later, but advanced filter may be better served by a JSON format that can be persisted.

ARK2/Development

2016-10-29T17:54:02Z

John Layt: /* Supported Platforms */

= Technical Details =

== Supported Platforms ==

ARK2 will only actively support platforms that are actively supported with security fixes by the upstream developers. ARK2 may run on platforms not actively supported, but this is not guaranteed and ARK2 may in the future require features only available in the supported platform versions.

* Apache is supported and will require mod_rewrite, other servers may work.
* PHP: PHP 7.4 is required due to new language features. The minimum version will be the minimum supported by PHP (see in [http://php.net/supported-versions.php PHP Supported Versions].
* PHP intl extension and ICU will be required, the opcache and APCu cache extensions are strongly recommended for performance.
* MySQL/MariaDB v5.7.8 or later (lowest supported MySQL, required for timestamps, full-text index, spatial index, and JSON type), utf8mb4 codeset only
* PostgreSQL v10 or later
* SQLite 3.9 or later (required for multiple inserts, full-text index, JSON type), Spatialite extension for spatial data
* Redis is recommended for data caching
* Linux and Mac are actively supported, Windows is also supported but minimally tested

== Browsers ==

ARK2 uses Bootstrap 3 for the default frontend. ARK2 therefore supports the browser versions supported by Bootstrap 3:

* Android 2.3
* Android >= 4
* Chrome >= 20
* Firefox >= 24
* Explorer >= 8
* iOS >= 6
* Opera >= 12
* Safari >= 6

Note that particular features provided by add-ons may not support this set of browsers. Due to the separation of the ARK2 frontend, you are free to implement your own frontend in any toolkit should you have other requirements.

== Standards ==

The [http://www.php-fig.org/psr/ PHP-FIG standards] will be used:
* [http://www.php-fig.org/psr/psr-1/ PSR-1] and [http://www.php-fig.org/psr/psr-2/ PSR-2] Coding Standards
* [http://www.php-fig.org/psr/psr-3/ PSR-3] Logging API Standard
* [http://www.php-fig.org/psr/psr-4/ PSR-4] Auto-Loading Standard
* [http://www.php-fig.org/psr/psr-5/ PSR-5] PHPDoc Standard Proposal
* [http://www.php-fig.org/psr/psr-6/ PSR-6] Caching API Standard
* [http://symfony.com/doc/current/components/http_foundation.html Symfony HTTP Foundation] for interchangeable Request/Response objects
* [http://www.php-fig.org/psr/psr-7/ PSR-7] HTTP Message Interface for interchangeable Request/Response objects if required for some components
* HTML5 / CSS3 ES vTBD
* All files will be UTF8 using UNIX LF

Plugins are available for Eclipse, Atom, and other IDEs to automatically check and/or apply these standards.

== Framework and Components ==

The chosen framework is the [http://silex.sensiolabs.org/ Silex] micro-framework built using [http://symfony.com/ Symphony] components, with a future upgrade path to the full Symfony Framework

Components will be carefully chosen to be well supported, stable, and interchangeable wherever possible. A number of criteria will be applied in selection:
* Must be standards compliant
* Must be well supported with a solid development history
* Must be well documented
* Must be widely used and supported
* Must have a strong community, small one-person efforts will only be considered if they are the de-facto standard or a 'well-known' developer
* Any database access must use Doctrine DBAL or ORM

Significant and reliable sources of components include:
* [https://packagist.org/ Packagist], the Composer index
* [http://symfony.com/components Symfony]
* [http://thephpleague.com/#packages PHP League]
* [https://zendframework.github.io/ Zend]
* [http://docs.sylius.org/en/latest/components/index.html Sylius], components from an e-commerce platform based on Symfony
* [http://auraphp.com/ Aura]
* [https://github.com/silexphp/Silex/wiki/Third-Party-ServiceProviders-for-Silex-2.x Silex providers]

== PHP Autoloading ==

PSR-4 is used for packaging, namespace and auto-loading of code.
* [https://getcomposer.org/ Composer] will be required for dependency management and PSR-4 auto-loading
* All external libraries will be installed by Composer under vendor/
* All code to be Object Oriented and implemented using a namespace of ARK
* All code will be under src/<namespace>

A good series of articles explaining PSR-4 and modern development and packaging in general can be found at the following:
* http://culttt.com/2013/01/07/what-is-php-composer/
* http://culttt.com/2014/03/12/build-php-package/
* http://culttt.com/2014/05/07/create-psr-4-php-package/

== Project Management ==

The ARK project is hosted on [https://gitlab.com/ GitLab], so you will need a free GitLab account to contribute to development. Bug reports may be opened using a Google or Facebook account. The code is also mirrored on GitHub.

The umbrella ARK project is organised under the [https://gitlab.com/arklab @arklab] group on GitLab, which includes the [https://gitlab.com/arklab/arkserver ARK Server] project. All public development and supported releases occur in this project repository. All code, issues, and wiki documentation on this project are public. All code contributions must pass formal code review before being merged. A separate GitLab group for [https://gitlab.com/lparchaeology L - P : Archaeology] manages private client forks and deployments of ARK Server.

GitLab provides a very powerful web application for managing projects, but most local development is done using the standard git command line tools. If you are new to Git, you may find a desktop application easier to use. Options include [https://www.sourcetreeapp.com/ SourceTree], [https://www.gitkraken.com/ GitKraken], and [https://desktop.github.com/ Github desktop].

=== Branching Strategy ===

We will use a simplified version of the [https://wiki.qt.io/Branch_Guidelines Qt git workflow].
* '''master''' branch is the semi-stable development branch for the next release which should always be in a stable enough state to pass CI testing
* New feature or bugfix branches for the next release are branched off '''master''' prefixed with the name of the author/group/client and including the feature or bug number, e.g. '''jlayt/linked-data''' or '''jlayt/issue-1234'''
* Once a new feature is finished and passes testing, it is rebased onto the current '''master''' and merged via a merge request with review
* Alpha and Beta testing releases will be tagged on '''master''' and not branched
* Once '''master''' is deemed feature complete and stable enough for a Release Candidate then a release branch will be branched off '''master''' with the minor release number, e.g. '''release/2.0''', '''release/2.1''', etc
* Further testing will take place on the release branch, with fixes applied to to the release branch as required
* Once ready for release, the release will be tagged with the point release number (e.g. 2.0.0) and all bugfixes merged back into '''master'''
* Any bugfixes for the stable release will be made in the earliest release branch they apply to and then merged forward through each release then eventually into '''master''' (merges may be immediate or periodic depending on the volume)
* Bugfix branches for the stable release are branched off the release branch prefixed with the name of the branch and including the feature or bug number, e.g. '''fix/2.1/issue-1234'''

For practical purposes during alpha development of ARK Server 2.0 a simplified approach will be used:
* '''dev''' branch is a temporary unstable development branch where code changes are developed
* '''master''' branch will occasionally be refreshed from '''dev''' with finished near-stable features

=== Custom Forks ===

The following approach is recommended for developing custom versions of ARK Server outside the main ARK Server project
* Create a new empty development repository
* Add the arklab/arkserver repository as a git remote
* Create a '''dev''' branch tracking a formal release branch on arklab/arkserver
* Create a '''test''' branch tracking the '''dev''' branch
* Create a '''prod''' branch tracking the '''test''' branch
* All custom code development occurs on the '''dev''' branch
* When custom code is stable for deployment to test server the '''dev''' branch is merged into the '''test''' branch
* When custom code is stable for deployment to production server the '''test''' branch is merged into the '''prod''' branch
* Production hotfixes can be developed on '''prod''' or a branch forked from '''prod'''
* When a new ARK Server version is released the '''dev''' branch is rebased onto the new remote release branch for testing and then merged forward for testing
* Test server deployment is a clone of the dev repository checked-out to the '''test''' branch and rebased when new test releases occur. Test config settings may be committed to the repo here.
* Production server deployment is a clone of the dev repository checked-out to the '''prod''' branch and rebased when new production releases occur. Prod config settings may be committed to the repo here.

The steps to set this up on GitLab are:
* Create the new project
* Create the dev branch tracking the arklab/arkserver release branch
* Create the test branch tracking the dev branch
* Create the prod branch tracking the test branch
* Protect the test and prod branches to allow only push/merge by Masters

The git commands required to set this up on a local server:
* mkdir <project>
* cd <project>
* git init
* git remote add arklab git@gitlab.com:arklab/arkserver.git
* git fetch arklab
* git branch -t dev arklab/master
* git branch -t test dev
* git branch -t prod test
* git checkout dev

== Folder Structure ==

The following install root folder structure is used by ARK2, based on the default Silex and Composer structure, but adapted to support multi-site and multi-tenant.

/
|- bin/
|- sysadmin
|- build/
|- <see [[ARK2/Frontend|Frontend]]>
|- sites/
|- src/
|- tests/
|- var/
|- cache/
|- log/
|- vendor/

* The 'bin' folder holds any executable binaries, primarily the sysadmin console script
* The 'build' folder holds tools and source files for building [[ARK2/Frontend|frontends]]
* The 'sites' folder holds the site specific files in a way that supports multi-site installs
* The 'src' folder holds the namespaced source code, e.g. src/ARK, src/MySite, etc
* The 'tests' folder holds the ARK auto-tests
* The 'var' folder holds install-wide transitory files, such as caches and logs (site-specific go in the site folder)
* The 'vendor' holds the component library code managed by Composer

Tarball packaging for release will not include the 'build', 'test', or 'vendor' folders.

The ARK2 'src' folder is organised as follows:

/
|- src/
|- ARK/
|- frontend/
|- api/
|- admin/
|- core/
|- server/
|- php/
|- schema/
|- database/
|- json/
|- <project>/
|- frontend/
|- <custom>/
|- server/
|- <custom>/

* The 'src/ARK' folder holds the ARK Project specific code, this code is not intended for custom site code and will be over-written by any ARK update
* The 'src/ARK/server' folder holds the ARK Server code, i.e. the backend code, API, database, etc
* The 'src/ARK/frontend' folder holds the ARK Web Front-end code. See [[ARK2/Frontend|Frontend]] for more details.
* The 'src/ARK/schema' folder holds the ARK Server data schemas, such as database table definitions, instance data models, etc
* The 'src/<project>' folder would hold custom code where another project wishes to run their own multi-site install with a single custom frontend, see [[ARK2/Frontend|Frontend]] for more details

The ARK2 'sites' folder is organised as follows:

/
|- sites/
|- <site>/
|- bin/
|- console
|- config/
|- credentials.json
|- database.json
|- site.json
|- files
|- <see [[ARK2/Files|Files]]>
|- schema/
|- templates/
|- <frontend>/
|- translations/
|- <frontend>/
|- var/
|- cache/
|- log/
|- web/
|- .htaccess
|- index.php
|- assets/
|- <frontend>/

* The 'sites/<site>' folder holds all the custom files required for a site. This allows for multi-site installs, as well as a simple site backup/transfer strategy. The <site> name is recommended to be the reverse domain name for the site if unique (e.g. com.example.www) or the subfolder if using a shared domain (e.g. mysite).
* The 'sites/<site>/bin' folder holds any site-specific executable binaries, primarily the site admin console script
* The 'sites/<site>/config' folder holds all the configuration files for the site.
* The 'sites/<site>/files' folder holds all the data files for the site, such as photos and documents. See [[ARK2/Files|Files]] for more details.
* The 'sites/<site>/schema' folder holds the site specific JSON Schema files.
* The 'sites/<site>/templates' folder holds the frontend template files.
* The 'sites/<site>/translations' folder holds the frontend translation files.
* The 'sites/<site>/web' folder is the webroot for the site, isolating the front-controller and assets in this folder improves security.
* The 'sites/<site>/web/assets' folder holds the assets for the frontend(s), usually as a symlink back to the 'src/ARK/web/<frontend>/assets' folder to allow for easy updates. See [[ARK2/Frontend|Frontend]] for more details.

=== IDE ===

While text editors and IDEs are a deeply personal choice, we recommend using Eclipse or [https://atom.io/ Atom] as they are cross-platform Open Source IDEs with powerful plugins to support the tools used by ARK. Using Eclipse or Atom ensures you have an environment consistent with the core ARK developers and ARK development standards.

ARK2/Database

2016-10-29T17:10:14Z

John Layt:

Note: This page is very outdated and pending revision.

= Database =

In ARK1, PDO is used to directly access only MySQL databases, and DB access statements are widely spread through the code base and manually coded. While PDO abstracts the connection, it doesn't abstract the SQL dialect so adding support for other databases such as Postgres or SQLite would require considerable work. It also makes migration to proper transaction support and performance improvements difficult, and is a security risk due to programmer error.

A Database Abstraction Layer (DAL) can abstract away the differences in SQL between database systems, and also provide Query Builders, Schema Management, and Migration tools to address the other issues. Most are built on PDO and can seamlessly integrate with legacy code to make for an easier migration path. The [http://doctrine-orm.readthedocs.io/projects/doctrine-dbal/en/latest/ Doctrine DBAL] DAL has been selected due to it's use in Symfony and the ORM extension being a Data Mapping ORM.

== Database Abstraction ==

Currently, PDO is used to directly access only MySQL databases, and DB access statements are widely spread through the code base and manually coded. While PDO abstracts the connection, it doesn't abstract the SQL dialect so adding support for other databases such as Postgres or SQLite would require considerable work. It also makes migration to proper transaction support and performance improvements difficult, and is a security risk due to programmer error. A Database Abstraction Layer (DAL) can abstract away the differences in SQL between database systems, and also provide Query Builders, Schema Management, and Migration tools to address the other issues. Most are built on PDO and can seamlessly integrate with legacy code to make for an easier migration path.

Longer term, full OO code, most frameworks, and many components use an ORM to map relational data to objects. A key part of choosing a framework or component eco-system is the ORM it uses. Most ORMs however use the Active Record pattern which cannot map onto the existing ARK data model. ARK would require a Data Mapper ORM to access the legacy database structure. While using multiple ORMs would be possible, it is not recommended due to performance overhead and potential contention.

[http://www.doctrine-project.org/ Doctrine ORM] is the only PHP Data Mapper available, and is built on the [http://doctrine-orm.readthedocs.io/projects/doctrine-dbal/en/latest/ Doctrine DBAL] DAL. Doctrine is widely use and under active development, being the main ORM for the Symfony eco-system as well as many independent components. DBAL also provides the full set of required Drivers, Query Builder, Schema Management and Migration tools to abstract access to the required databases.

Automated database creation and upgrade will be implemented using Doctrine Migrations:
* The core database schema will be defined in [http://andrewembler.com/2015/05/describe-parse-and-utilize-database-schemas-doctrine-xml/ doctrine-xml] (possibly initially reversed engineered from existing v1.x database). This schema will be stored in the build tools folder and will not be deployed.
* A custom Doctrine\DBAL\Migrations\Provider\SchemaProvider class will use the doctrine-xml schema to generate the full schema for new databases.
* A custom Doctrine\DBAL\Migrations\Provider\SchemaProvider class will use the doctrine-xml schema to generate schema diff to automatically create the Doctrine Migration class required for each version upgrade.
* The admin console will provide the standard migration tools (diff perhaps only from build console?)
* The ARK install script will call the doctrine tools to create the database
* If outstanding migrations are found after an upgrade, then the site will go into maintenance mode

The creation of tables required for custom modules will be provided via API and not via the Migrations. This will be part of the data schema creation code and not the core database code.

Multiple database connections within an ARK will be supported, with the database to be used by any data model passed in using dependency injection. This will improve the code used for import, export and migration. It will also allow for separate databases for the admin (user, config, etc) and the data which may assist with AaaS and Multi-tenancy.

Database security will be tightly controlled to prevent security breaches in the AaaS model. There will be four levels of database account:
* Database Admin - Full admin account, credentials never stored on webserver, asked for by install script
* AaaS Admin - Limited admin account for AaaS client admin, only allowed to create new database and add required users, credentials never stored on webserver, passed in to install script by portal script
* ARK Admin - Limited admin account for ARK instance admin (create module tables only?), created by install script, credentials never stored on server, requested from user when needed (create modules, etc)
* ARK User - Limited user account for read/write data access only, created by install script, credentials stored on webserver

== ID Generation ==

A number of possible strategies exist for the allocation of Item numbers, which must be performed in a way abstracted from the database server used, and allowing for the use of offline mobile devices.
* Monotonically increasing sequence within any parent (default)
* Monotonically increasing sequence within any parent and within a predefined range determined by some compulsory attribute of the object, e.g. Trench Number
* Manually allocated IDs in any sequence, but still with the option for default monotonic IDs

Due to the cascading of the generated ID throughout the database structure, the ID needs to be pre-allocated before the update can be performed. However, this risks leaving IDs allocated but unused should the actual update transaction be rolled back. The update to the sequence cannot be reversed, as it may have been further updated by another transaction. Instead such orphans will need to be tracked for later recycling.

Taking all these factors into account, the ID Allocation will function as follows:
* On creation of a new Item, the Entity Manager will be called to create a new ID
* The EM will decide which sequence to use for the ID
* The EM creates a new update transaction on the database and looks in the sequence lock table for any IDs to be recycled
* If there is one to be recycled, it is updated to locked and the ID returned
* If there are none to be recycled, then the EM updates the sequence in the sequence table, creates a new sequence lock entry, and returns the allocated ID
* When the EM performs the actual persistence of the object, then the sequence lock will be deleted
* If the insert transactions fails and is rolled back, then the EM will update the sequence lock to be recycled

For offline mobile use, there are two possible strategies:
* Where the ID carries no inherent meaning, or is not used in the field, then a temporary hidden ID can be used until sync when real IDs are allocated using the standard method above
* Where the ID has meaning and is used in the field, such as numbering context bags or sample buckets, then the real IDs will have to be pre-allocated to the mobile device

The API will provide a method for a mobile device to 'check-out' a block of numbers for use, and the ability to check them back in again:
* The device requests a new block of IDs
* The EM creates a new update transaction on the database, updates the sequence table, and creates an entry in the sequence reserve table for the allocated range with a token which is returned to the device
* The device uses the allocated range when creating new items
* If the device runs out of IDs the user is warned that subsequent IDs may be duplicated
* On syncing, the device tells the API what ID it allocated and the token of the reserved sequence
* On inserting, the EM uses the sequence reserve table instead of the main sequence table to track the IDs
* If the device attempts to use IDs outside the reserved sequence then and those IDs are free, then they will be allowed, otherwise the updates will be rejected/quarantined
* When the sync is finished, the user will be asked if they want to recycle any left-over IDs, or if they want more allocated

== Chains ==

Chains are a technique in ARK for storing hierarchical tree data in relation form. This is done using an adjacency table method. This is a problem in ARK2 for a number of reasons:
* Knowledge of when data is stored in chains is held solely in the subform code that creates or reads the chain, which causes issues when the schema will need to represent the data structure without the subform
* Chains are an internal implementation detail, their existence should not 'leak' into the schema or api for external clients, they must be free to choose their own storage solution
* Access to chains can be slow and inefficient, especially walking down a tree when you don't know how 'wide' it is (i.e. what data fragments it has as descendants).

The data schema will not represent data as chains, trees or graphs. Instead the schema will merely represent the inherent hierarchical structure of the data using groups/objects/lists. The data persistence layer will know to persist those repeating groups in its model. In the case of the current ARK SQL database this means repeating groups will need to be stored as chains.

A number of other techniques exist to make reading/writing of trees faster, such as Nested Sets. The Closure Table technique has been chosen for a number of reasons:
* Is a simple extension to the existing Adjacency Table method
* Is fast to both read and write
* An entire tree can be read with a single SQL query
* Supports storing DAGs, i.e. Harris Matrix

The proposed implementation will be:
* Chains may be renamed as Graphs?
* Nodes and Edges are stored separately
* The Nodes in a Graph are either Items or Fragments, but not both
* The Nodes continue to be stored in their current tables
* Edges are stored in a new ark_data_graph table
* Item Graphs store their itemkey/itemvalue Edges in the ark_data_graph table
* Fragment Graphs store their table/id Edges in the ark_data_graph table
* Descendent Fragment Nodes will no longer be keyed by their direct parent table/id, but will instead be keyed by their root itemkey/itemvalue. This allows faster access to all nodes, and means the nodes never need to be updated whenever the graph is altered.
* Root Fragment Nodes may need to carry a flag to indicate they are a root, otherwise a lookup is required every time on the graph table.

New code will be needed to create and maintain graphs in place of the current chain code. Migration code will be required to build the graph for existing data and update the nodes.

References:
* SQL Anti-patterns book
* http://technobytz.com/closure_table_store_hierarchical_data.html
* http://people.apache.org/~dongsheng/horak/100309_dag_structures_sql.pdf
* http://timwi.blogspot.co.uk/2010/03/query-tree-structures-and-dags-in-sql.html
* http://dirtsimple.org/2010/11/simplest-way-to-do-tree-based-queries.html
* http://mikehillyer.com/articles/managing-hierarchical-data-in-mysql/
* https://github.com/Atlantic18/DoctrineExtensions/blob/master/doc/tree.md
* https://github.com/EspadaV8/ClosureTable

ARK2/Localization

2016-10-29T16:46:30Z

John Layt: /* Localization */

= Localization =

Localization is the process of translating the user interface and data into the language and formats of the user.

ARK2 uses the following components for localization and translation:
* The [http://symfony.com/doc/current/components/intl.html Symfony Intl] component as a wrapper for the [http://php.net/manual/en/book.intl.php PHP Intl] extension and ICU, for collation, formatting and parsing
* The [http://symfony.com/doc/current/components/translation.html Symfony Translation] component for translation
* The standard JS [https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Intl Intl API], or the [https://github.com/andyearnshaw/Intl.js Intl Polyfill] where Intl is not available

== Translation ==

* Translations are stored in the core database
* Translations are exported into XLIFF file format, allowing offline apps to download the full catalogue
* Translations uses keys rather than source strings
* Translations will be split in a number of catalogues:
** Vocabulary
** Core interface
** User interface
** Admin interface
** Schema (one for each created)
* Actor and User IDs are translated live using a custom loader from the database

The Admin Interface provides a basic interface for the maintaining of translation. Export and import interfaces are provided for the chosen online translation community.

== Translation Workflow ==

Translation workflow will have two modes, Debug and Live.
* In Debug mode, the database is the canonical source for translations
* In Live mode, the XLIFF files are the canonical source

In Debug mode:
* Translations can be disabled in the site.json file for testing or development purposes
* Translations are loaded live from the database
* Translation Admin page shows stats, allows editing of translations, and import/export to XLIFF (also to translation website)
* Translation extractors can be run to extract the used keys from code files
* Translations can be edited in the Translation Profiler page
** Profiler bar shows translation stats for the page
** Profiler page allows live editing of translations used on the page with persistence back to the database and flagging as fuzzy

In Live mode:
* Translations are always active, sites.json disable is ignored
* Translations are loaded from the XLIFF files, database is only used for Actors/Users
* Translation Admin page is available, but changes do not take effect until exported to XLIFF

== Development Required ==

No existing package supports our requirements so we will need to fork a number of existing projects (see below):
* Translation extractors
* Translation admin page
* Translation Profiler page

== Design Decisions To Be Made ==

Potential online portals include:
* [https://www.transifex.com Transifex] - Market leader, but now closed source, offers free hosting to open source projects, large community, great features and automated workflow, API, Github integration, etc.
* [http://zanata.org/ Zanata] - A Red Hat open-source project (in response to Transifex going closed) with free hosting and large existing community, or can be self-hosted (JBoss based), great features and automated workflow, API, Github integration, etc. No RTL?
* [https://weblate.org/en/features/ Weblate] - An open-source project with free and paid-for hosting, but no central community, or can be self-hosted, good features, API, Github integration, etc. Django based.
* https://localise.biz/ Free or GBP5 p/m plan may be enough, good api, Symfony bundle
* [http://translationproject.org/html/welcome.html Translation Project] - A FSF open-source project with free hosting and large existing community, but very basic features and manual workflow.
* [http://pootle.translatehouse.org/index.html Pootle] - An open-source project, self-hosted, Django based

Zanata would seem the preferred option for a hosted service, but Weblate might be better if self-hosting.

A number of options exist for automating extraction of Symfony translations, or for allowing interactive editing inside the admin panel or profiler panel:
* https://jolicode.com/blog/translation-workflow-with-symfony2
* https://developer.happyr.com/how-happyr-work-with-symfony-translations
* https://github.com/deanc/silex-web-translator
* https://github.com/Aecf/TranslatorToolBundle
* https://github.com/instaclick/TranslationEditorBundle
* https://github.com/lexik/LexikTranslationBundle
* https://github.com/manuelj555/ManuelTranslationBundle
* https://github.com/Happyr/TranslationBundle
* http://jmsyst.com/bundles/JMSTranslationBundle

None of these quite match our requirements, but may be adapted to achieve our workflow.

Translations in javascript:
* https://github.com/willdurand/BazingaJsTranslationBundle
* https://github.com/wikimedia/jquery.i18n

QGIS/ARK Grid

2016-08-09T13:06:27Z

John Layt: /* ARK Grid */

= ARK Grid =

A QGIS Plugin to create and work with local site grids as commonly used in archaeology.

A grid wizard allows you to create grids from two known points, or from a known origin and baseline. Interactive tools are provided to convert between local coordinates and map coordinates.

[[File:ark_grid.png|center|link=|ARK Grid - Panel]]

For details on installation, support, licensing, and development, please see the main [[QGIS|ARK QGIS plugins page]]

== Settings Wizard ==

To use ''ARK Grid'' click on the ''ARK Grid'' icon in the main toolbar.

[[File:grid_icon.png|center|link=|ARK Grid - Logo]]

If you have not run ''ARK Grid'' before then the ''Settings Wizard'' will be run to set up the required grid layers. Your chosen settings will be written into your current QGIS project, you must save your project after the wizard has run otherwise your settings will be lost. If you do not save your settings then the wizard will run again and you will be able to setup where your layers are already saved.

[[File:grid_wizard1.png|center|link=|ARK Grid - Project Wizard Page 1]]

Choose where your project home folder is located. If your QGIS project already has a project home folder set then this will be shown as the default folder. The ''ARK Grid'' layers will be saved in the Project Folder in a subfolder 'vector/grid'.

[[File:grid_wizard2.png|center|link=|ARK Grid - Project Wizard Page 2]]

Choose what you want to name the grid layers and layer group, recommended default names are provided. You must have a grid point layer to use for the calculations, but the line and polygon layers are optional.

[[File:grid_wizard3.png|center|link=|ARK Grid - Project Wizard Page 3]]

On clicking Done the required folders and files will be created if they do not already exist. Existing files will '''never''' be overwritten. The layers will then be added to the project ''Layers Panel'' and the ''ARK Grid Panel'' will be displayed. Only the ''Create New Grid'' tool will be enabled at this stage.

[[File:grid_layers.png|center|link=|ARK Grid - New Layers]]

[[File:grid_panel_new.png|center|link=|ARK Grid - New Panel]]

== New Grid Wizard ==

To create a new grid, click on the ''Create New Grid'' tool.

[[File:grid_new_icon.png|center|link=|ARK Grid - Create New Grid Icon]]

[[File:grid_wizard_new1.png|center|link=|ARK Grid - New Grid Wizard Page 1]]

[[File:grid_wizard_new2.png|center|link=|ARK Grid - New Grid Wizard Page 2]]

[[File:grid_wizard_new3.png|center|link=|ARK Grid - New Grid Wizard Page 3]]

[[File:grid_wizard_new4.png|center|link=|ARK Grid - New Grid Wizard Page 4]]

[[File:grid_new.png|800px|center|link=|ARK Grid - New Grid]]

== Grid Tools ==

Once you have created a valid grid, all the ''Grid Tools'' will be enabled.

[[File:grid_panel_full.png|center|link=|ARK Grid - Grid Panel]]

; Current Grid Dropdown
: If you create more than one grid, you can switch between grids using the grid selection dropdown box. Only the selected grid is displayed and used in calculations.

[[File:grid_grid_combo.png|center|link=|ARK Grid - Choose Grid Combo]]

; Map Point Entry
: The ''Map Point'' input boxes display the currently calculated Map Point. You can manually enter a Map Point and the Local Point will be automatically updated.

[[File:grid_map_entry.png|center|link=|ARK Grid - Map Point Entry]]

; Local Point Entry
: The ''Local Point'' input boxes display the currently calculated Local Point. You can manually enter a Local Point and the Map Point will be automatically updated.

[[File:grid_local_entry.png|center|link=|ARK Grid - Local Point Entry]]

; [[File:grid_tool_icon.png|link=|ARK Grid - Identify Grid Coordinates Tool]] Identify Grid Coordinates Tool
: The ''Identify Grid Coordinates'' tool allows you to click on the map and have the Map Point and Local Point coordinates displayed for the point clicked on.

; [[File:grid_pan_icon.png|link=|ARK Grid - Pan Map Tool]] Pan Map Tool
: The ''Pan Map'' tool will pan the map to the currently entered Map Point.

; [[File:grid_paste_icon.png|link=|ARK Grid - Paste Map Point Tool]] Paste Map Point Tool
: The ''Paste Map Point'' tool will paste a Map Point in WKT format from the clipboard. This can be used to copy a point from a layer in QGIS.

; [[File:grid_point_icon.png|link=|ARK Grid - Add Point Tool]] Add Point Tool
: The ''Add Point'' tool adds the current Map Point to the currently selected layer if it is a vector point layer in editing mode.

; [[File:grid_update_icon.png|link=|ARK Grid - Update Layer Coordinates Tool]] Update Layer Coordinates Tool
: The ''Update Layer Coordinates'' tool shows a dialog to let you update a point vector layer to either add the local coordinates as attribute fields or to set the map point from local coordinate attribute fields.

; [[File:grid_settings_icon.png|link=|ARK Grid - Run Settings Wizard Tool]] Run Settings Wizard Tool
: The ''Run Settings Wizard'' tool will run the settings wizard.

; [[File:grid_copy_icon.png|link=|ARK Grid - Copy Point Tool]] Copy Point Tool
: The ''Copy Point'' tool will copy the Map Point or Local Point to the clipboard in WKT format.

== Layer Styles ==

The plugin ships with a default set of style files for the grid layers. If you wish to modify the styles, simply save your modified style files into your grid folder with the same names as your grid layers.

QGIS

2016-08-09T11:11:09Z

John Layt: /* Setup */

= QGIS =

[http://www.qgis.org/en/site/about/index.html QGIS] is a powerful, user friendly, free and Open Source Geographic Information System (GIS). It runs on Windows, Mac OS X, Linux, Unix, and Android and supports numerous vector, raster, and database formats and functionalities.

At L ~ P : Archaeology we exclusively use QGIS for all our GIS requirements and have developed a number of plugins to support archaeological projects and to integrate with ARK Database. We have released some of these plugins for all archaeologists to benefit from and will release more over time.

Currently these plugins are hosted in our beta testing repository and are marked as experimental, but once stable will be submitted to the main QGIS repository. You are welcome to try out these plugins and to provide feedback, but please be aware that they are in beta and so may still have some bugs. To provide feedback, either open an issue on the plugin's GitHub issue tracker, or email ark@lparchaeology.com.

== Development ==

Development of these plugins is sponsored by [http://www.lparchaeology.com/cms/ L ~ P : Archaeology] using the Free Software philosophy and Open Source development methodologies. All of the code is developed in the open on [https://github.com/lparchaeology/ GitHub] and contributions from other interested parties are always welcome. If you would like to commission or sponsor development of new features or plugins then please contact us.

== Support ==

The plugins offered here are in beta and no warranty or support is offered for them. That said, we are keen to receive feedback on the plugins and will respond as best we can. We plan to set up appropriate community support channels in the future. In the meantime, you can raise issues via the appropriate GitHub page, or email us at ark@lparchaeology.com.

== Licence ==

All L ~ P : Archaeology plugins are developed under the [https://simple.wikipedia.org/wiki/GNU_General_Public_License GPL2 licence], the same as QGIS itself and all plugins in the main QGIS repository. The GPL allows you to freely use and modify the plugin as you wish, provided you share with the community any improvements to the code you might distribute. Share and share alike, it's only fair.

== Setup ==

To install the plugins you need to add the L ~ P : Archaeology QGIS Plugins repository in your QGIS plugins settings.

* From the ''Plugins'' menu choose ''Manage and Install Plugins''
* In the ''Plugins'' window choose the ''Settings'' tab
* Select the ''Show also experimental plugins'' option
* Select the ''Add...'' button
* Add the new repository details with the name ''ARK Plugins'' and a URL of ''https://plugins.lparchaeology.com/qgis/plugins.xml''
* After clicking ''OK'' the repository will be loaded and the new plugins will be listed in the ''New'' tab

The ''Plugins Settings'' dialog should look similar to this before adding the ARK repository:

[[File:qgis_plugins.png|600px|center|QGIS Plugin Settings]]

The ''Repository Details'' dialog should look similar to this:

[[File:qgis_repo.png|center|QGIS Repository Settings]]

== Available Plugins ==

The following plugins are currently available in the repository.

=== ARK Clone ===

A QGIS Plugin to clone existing vector layers including styles as shapefiles or temporary memory layers.

This plugin adds an option to the legend context menu and the main vector menu to clone the current layer. Users can choose to clone just the layer structure, just the selected features, or all the features.

Development of ''ARK Clone'' is hosted on GitHub at https://github.com/lparchaeology/ArkClone. You can open an issue at https://github.com/lparchaeology/ArkClone/issues, or email ark@lparchaeology.com.

=== ARK Grid ===

A QGIS Plugin to create and work with local site grids as commonly used in archaeology.

A grid wizard allows you to create grids from two known points, or from a known origin and baseline. Interactive tools are provided to convert between local coordinates and map coordinates.

[[File:ark_grid.png|center|ARK Grid Panel]]

Detailed documentation can be found on the [[QGIS/ARK_Grid|ARK Grid page]].

Development of ''ARK Grid'' is hosted on GitHub at https://github.com/lparchaeology/ArkGrid. You can open an issue at https://github.com/lparchaeology/ArkGrid/issues, or email ark@lparchaeology.com.

== Future Plugins ==

The following plugins are currently under development.

=== ARK Snapping ===

A QGIS Plugin to speed-up snapping configuration.

This plugin is currently in alpha and will be released as beta shortly.

Main Toolbar buttons allow you to quickly switch between All Layers, Current Layer, and Selected Layers, and to toggle Intersection Snapping and Topological Editing:

[[File:ark_snap_main.png|center|ARK Snapping Toolbar Buttons]]

When in Selected Layers mode, the Layer Context Menu allows you to configure snapping on the selected layers.

[[File:ark_snap.png|center|ARK Snapping Layer Context Menu]]

In Selected Layers mode, a new panel will list all selected layers and allow for more detailed configuration. This has yet to be implemented.

A custom version of the panel is implemented in ARK Spatial, the action buttons for each layer will be changed to a list of layer names, with buttons to add/remove available layers.

[[File:ark_snap_panel.png|center|ARK Snapping Layer Panel]]

=== ARK Spatial ===

A QGIS Plugin to record and query archaeological spatial data.

This plugin is currently in alpha and will be released as beta shortly.

Features include:
* Simplified geo-referencing of permatrace to a site grid or baseline
* Rapid digitising of archaeological plans and sections using MoLAS drawing conventions
* Powerful query and filter tools working at Context level
* Post-excavation checking workflow and reports
* Live data link to an ARK Database

[[File:ark_spatial.png|800px|center|ARK Spatial]]

=== ARK Georeferencer ===

A QGIS Plugin to georeference permatrace to a site grid.

This plugin is currently part of ARK Spatial but may be released as a separate plugin.

[[File:ark_georef.png|800px|center|ARK Georeferencer]]

== Library for Plugin Developers ==

Many common functions and classes needed in the development of QGIS plugins have been aggregated into a common library [https://github.com/lparchaeology/libarkqgis libarkqgis] that is reused by all our plugins. Other plugin developers may find this library useful, although we do not provide a stable API guarantee as yet or a even a stable release branch.

ARK2

2016-02-03T12:01:50Z

John Layt: /* Aims */

This page details the progress on development of ARK 2.0

== Aims ==

The primary aims of ARK2 are:
* Full code re-write to modern standards using modern components
* Separate the ARK Database backend from the ARK Web frontend
* Implement a modern REST API to allow other frontends and apps to access and update the ARK Database
* Simplify the setup and configuration of ARK by moving the config into the database and providing online config tools
* Improve the overall performance and data integrity of ARK
* Make it possible to provide an ARK hosting service

Modern frontend
* HTML5
* Bootstrap based
* Twig templates
* Front Controller model
* Config driven pages views

Modern backend
* Full REST API to access and update all ARK data
* Database abstraction and Object Relational Mapping
* Config driven data schema
* Controlled Vocabularies and Linked Open Data
* User Authentication via internal user/password and external OAuth2 providers (Facebook, Google, etc)
* User Authorisation via Role Based Access Control (RBAC) using hierarchical Roles and Permissions structure
* Field-level data access control
* Data Workflows in conjunction with User Authorisation control

== Documentation ==

Details of ARK2 can be found in the following sections:

* [[ARK2/Design|Design]] - High Level Design Decisions
* [[ARK2/Technical|Technical]] - Technical details, Development tools and procedures
* [[ARK2/The_ARK_Way|The ARK Way]] - Web Development - The ARK2 Way
* [[ARK2/Install|Install]] - Installation Instructions
* [[ARK2/Architecture|Architecture]] - System Architecture
* [[ARK2/Database|Database]] - Database / ORM details
* [[ARK2/Cache|Cache]] - Cache details
* [[ARK2/Model|Model]] - Data model / Schema
* [[ARK2/View|View]] - Views on the Data Model
* [[ARK2/Spatial|Spatial]] - Spatial data
* [[ARK2/Vocabulary|Vocabulary]] - Controlled Vocabularies
* [[ARK2/Files|File Management]] - File, Image, and other Media Management
* [[ARK2/Localization|Localization]] - Internationalization, Localization, and Translation
* [[ARK2/Security|Security]] - Security, Authentication, Authorisation, User Management
* [[ARK2/API|API]] - REST API implementation
* [[ARK2/Frontend|Frontend]] - Web Frontend
* [[ARK2/Templates|Templates]] - Using Twig for the Web Frontend
* [[ARK2/Console|Console]] - Admin Consoles
* [[ARK2/Admin|Admin]] - Admin Frontend
* [[ARK2/Branding|Branding]] - Branding of the ARK Project, Products, and Service

Sf chaintable

2015-10-12T16:24:37Z

Mike: /* Example Configuration */

===Description===

sf_chaintable can be used for displaying and editing data fragments that are chained to other data fragments. One 'root' field is linked to the item being viewed, which may have one or more fragments per item creating rows for the table and then one or more fragments can be chained to these to create additional columns. Examples of this include attaching numbers to find attributes - mirroring and supplements [[sf_assemblage]] for linking usage to periods of sites, or for levels tables.

By default the chaintable hides any empty columns. As of version 1.2 the new records feature is incomplete and only some of the edit functions are supported - numbers, text and attributes.

===Additional Fields===

The sf_assemblage needs a number of different 'op' variables set in the sf_conf:
*'''op_assemblage_type''' - the root field which is connected to the item
*'''op_sum_field''' - if this is set then the rows will be aggregated where they are identical and the number in this field will be summed for the output. this field must be a number!
*'''fields''' - the columns that will be added to the table

===Example Configuration===
<pre>
$conf_mcd_assemblage =
array(
'view_state' => 'max',
'edit_state' => 'view',
'sf_title' => 'assemblage', //appears in the titlebar of the subform (mk nname)
'sf_html_id' => 'pottery_ass', //the form id tag (must be unique)
'sf_nav_type' => 'nmedit',
'script' => 'php/subforms/sf_chaintable.php',
'op_assemblage_type' => $conf_field_sherd_class,// the field of data attached to the item
'op_sum_field' => $conf_field_total,
'default' => '0', //the default value for creating new records
'op_label' => 'space',
'op_input' => 'save',
'fields' => array(
$conf_field_sherd_class,
$conf_field_sherd_form,
$conf_field_sherd_type,
$conf_field_finddates,
$conf_field_sherdfunction,
$conf_field_catxmiloc,
$conf_field_total,
),
);
</pre>

[[category: Administrator]]

Importing Text Data

2015-09-14T08:44:07Z

Mike: /* Step 1. Identify datafile */

==ARK textfile importer user guide==
Since v1.1.2 ARK has had a tool to import text based files in either JSON or CSV.
.The file can be filtered so that only certain records are uploaded. It can create new records or add to existing ones.

===Step 1. Identify datafile===
[[File:jsonhowto_fig1.png|thumb|400px|right|Figure 1: File Picker]]
;Specify the location of the file.
:The import page of ARK in 1.1.2 includes a function for importing data directly from a text file or a data stream available online. Currently the file must be available on a web server. (Fig. 1) Enter the location of the file, either a relative path from the ARK root eg data/uploads/your_data.csv or a remote web address eg http://www.example.com/ARK/data?format=json.
Click Submit.

Depend on the size of the file it may take some time to load into your browser. Once it is loaded it will be available to manipulate very quickly.A spinner will appear while this is happening.

===Step 2. Determine 'Root' of the data structure.===

The data you have will likely include information that relates to several ARK items. Within your data structure there will be a level that includes all of these item representations - in the same way as rows in a table. In a csv file where there is a single record per row the root will be 'root>items'. In a more complicated json file (for example one with file level metadata held in the root as well as the items) the records may be contained in an object within the root object. Json allows for extensible representation, so each 'row' in the json file may not include all the same fields. It may be necessary to use the filter tools to only import objects which have a certain criteria.

The example shown here has many items in the items object in root. The first available column is used to identify each of the contained objects - in this case it is an ARK ID, but it could be anything and may not be unique if it is not unique in your structure. For example if field 1 in your table is a site code or context type.

[[File:jsonhowto_fig2.png|thumb|400px|left|none|Figure 2: Objects in 'root>items']]

In order to define this level you will need to navigate to it.The path at the top of the page describes where in the data structure you currently are.The large panel below the advanced options shows the identifiers held in the current level.At the root level of a CSV file the first column will be used as an identifier. Clicking on these will open the data attached to that identifier for viewing.(Fig. 2) It is possible to go up one level using the button at the end of the list of objects, or any part of the location at the top of the page can be used to navigate to that level.When the main window would contain more than 20 items it is truncated and only the first 20 are shown.

Use the 'this is root' button in the navigation panel at the top of the page when the root is displayed in the main panel.

===Step 3. Determine ARK IDs===

[[File:jsonhowto_fig3.png|thumb|right|600px|Figure 3: Root and ARK ID specified]]

Using the navigation method above find the path to the ARK id within your data. This will often be as simple as 'root>items>item1>Ark_id'.The object that you define the ARK ID in is not important. As long as the objects are structured consistently the ARK IDs will be retrieved automatically based on the root defined in the step above. so for instance the importer will import the ARK ID for METSUR_152 from root > items > METSUR_152 > ARK_id and then the ARK ID for METSUR_153 from root > items > METSUR_153 > ARK_id, and so on.

If you do not have ARK IDs in your data refer to the [[#Advanced Options|advanced options]].
Otherwise, click 'This is ARK ID' button below the identifier that contains the ARK ID.

===Step 4. Test Import===

[[File:jsonhowto_fig4.png|thumb|right|400px|Figure 4: Example of dry run]]

Click 'Import this' below the identifier that you would like to import. If you have completed the steps above a new panel will appear below the viewer. Confirm that this looks how you expect – '''this is what will be imported into the database'''.

The final panel allows you to define which records will be imported. A type must be specified. If you are importing a file it may be necessary to do a separate import of each type. This is because different types have different fields attached to them, the drop down menus respond to the ones above them, presenting only the options available. If they are not completed in order it may result in unexpected behaviour.

===Step 5.===
When you are happy that you are importing the correct data into the correct field, click 'Submit'.
You will be shown the results of your import. Click 'Import Json' to repeat the process for any other fields you wish to import.

==Advanced Options==
;Ste_cd:
:The site code that imported items will have, is one is not specified in the data – uses the ARK default if not set. No site code: This check box will remove site codes from the import itemvalue, for use with chains.
;Regular Expression:
:This is used to extract the itemvalue from the ARK ID column – by default it grabs the first run of numbers in the field.
;Start at Arbitrary Number:
:If you wish to create a sequence of numbers starting from a given point, (including 1) you will need to specify this here, and the number to start on. A unique 'ARK ID' must still be specified in the data, but it will not be used.

Failed to find markup

2015-08-28T11:33:29Z

Mike: Created page with "This is a common error message in ARK, don't panic, it simply means that the text fragment for this part of the ARK is not available in either the default or your chosen language..."

This is a common error message in ARK, don't panic, it simply means that the text fragment for this part of the ARK is not available in either the default or your chosen language.

Often you can make an educated guess at what it might be from the internal nickname the text has, which is also displayed in the error message.

If you have admin privileges you can add the missing markup in the 'Markup tab' see [[Markup Administration]] for help with that.