https://ark.lparchaeology.com/wiki/api.php?action=feedcontributions&user=Mike&feedformat=atomARK - User contributions [en]2024-03-28T15:35:57ZUser contributionsMediaWiki 1.27.1https://ark.lparchaeology.com/wiki/index.php?title=Importing_Text_Data&diff=124908Importing Text Data2018-04-03T12:17:55Z<p>Mike: /* Step 1. Identify datafile */</p>
<hr />
<div>==ARK textfile importer user guide==<br />
Since v1.1.2 ARK has had a tool to import text based files in either JSON or CSV. <br />
.The file can be filtered so that only certain records are uploaded. It can create new records or add to existing ones.<br />
<br />
===Step 1. Identify datafile===<br />
[[File:jsonhowto_fig1.png|thumb|400px|right|Figure 1: File Picker]]<br />
;Specify the location of the file.<br />
:The import page of ARK in 1.1.2 includes a function for importing data directly from a text file or a data stream available online. Currently the file must be available on a web server. (Fig. 1) Enter the location of the file, either a relative path from the ARK root eg data/uploads/your_data.csv or a remote web address eg http://www.example.com/ARK/data?format=json.<br />
Click Submit.<br />
<br />
Depend on the size of the file it may take some time to load into your browser. Once it is loaded it will be available to manipulate very quickly.A spinner will appear while this is happening.<br />
<br />
===Step 2. Determine 'Root' of the data structure.===<br />
<br />
The data you have will likely include information that relates to several ARK items. Within your data structure there will be a level that includes all of these item representations - in the same way as rows in a table. In a csv file where there is a single record per row the root will be 'root>items'. In a more complicated json file (for example one with file level metadata held in the root as well as the items) the records may be contained in an object within the root object. Json allows for extensible representation, so each 'row' in the json file may not include all the same fields. It may be necessary to use the filter tools to only import objects which have a certain criteria.<br />
<br />
The example shown here has many items in the items object in root. The first available column is used to identify each of the contained objects - in this case it is an ARK ID, but it could be anything and may not be unique if it is not unique in your structure. For example if field 1 in your table is a site code or context type.<br />
<br />
[[File:jsonhowto_fig2.png|thumb|400px|left|none|Figure 2: Objects in 'root>items']]<br />
<br />
In order to define this level you will need to navigate to it.The path at the top of the page describes where in the data structure you currently are.The large panel below the advanced options shows the identifiers held in the current level.At the root level of a CSV file the first column will be used as an identifier. Clicking on these will open the data attached to that identifier for viewing.(Fig. 2) It is possible to go up one level using the button at the end of the list of objects, or any part of the location at the top of the page can be used to navigate to that level.When the main window would contain more than 20 items it is truncated and only the first 20 are shown.<br />
<br />
Use the 'this is root' button in the navigation panel at the top of the page when the root is displayed in the main panel.<br />
<br />
===Step 3. Determine ARK IDs===<br />
<br />
[[File:jsonhowto_fig3.png|thumb|right|600px|Figure 3: Root and ARK ID specified]]<br />
<br />
Using the navigation method above find the path to the ARK id within your data. This will often be as simple as 'root>items>item1>Ark_id'.The object that you define the ARK ID in is not important. As long as the objects are structured consistently the ARK IDs will be retrieved automatically based on the root defined in the step above. so for instance the importer will import the ARK ID for METSUR_152 from root > items > METSUR_152 > ARK_id and then the ARK ID for METSUR_153 from root > items > METSUR_153 > ARK_id, and so on.<br />
<br />
<br />
If you do not have ARK IDs in your data refer to the [[#Advanced Options|advanced options]].<br />
Otherwise, click 'This is ARK ID' button below the identifier that contains the ARK ID.<br />
<br />
===Step 4. Test Import===<br />
<br />
[[File:jsonhowto_fig4.png|thumb|right|400px|Figure 4: Example of dry run]]<br />
<br />
Click 'Import this' below the identifier that you would like to import. If you have completed the steps above a new panel will appear below the viewer. Confirm that this looks how you expect – '''this is what will be imported into the database'''.<br />
<br />
The final panel allows you to define which records will be imported. A type must be specified. If you are importing a file it may be necessary to do a separate import of each type. This is because different types have different fields attached to them, the drop down menus respond to the ones above them, presenting only the options available. If they are not completed in order it may result in unexpected behaviour.<br />
<br />
===Step 5.===<br />
When you are happy that you are importing the correct data into the correct field, click 'Submit'.<br />
You will be shown the results of your import. Click 'Import Json' to repeat the process for any other fields you wish to import.<br />
<br />
==Advanced Options==<br />
;Ste_cd:<br />
:The site code that imported items will have, is one is not specified in the data – uses the ARK default if not set. No site code: This check box will remove site codes from the import itemvalue, for use with chains.<br />
;Regular Expression:<br />
:This is used to extract the itemvalue from the ARK ID column – by default it grabs the first run of numbers in the field.<br />
;Start at Arbitrary Number:<br />
:If you wish to create a sequence of numbers starting from a given point, (including 1) you will need to specify this here, and the number to start on. A unique 'ARK ID' must still be specified in the data, but it will not be used.</div>Mikehttps://ark.lparchaeology.com/wiki/index.php?title=Importing_Text_Data&diff=124907Importing Text Data2018-04-03T12:15:57Z<p>Mike: /* ARK textfile importer user guide */</p>
<hr />
<div>==ARK textfile importer user guide==<br />
Since v1.1.2 ARK has had a tool to import text based files in either JSON or CSV. <br />
.The file can be filtered so that only certain records are uploaded. It can create new records or add to existing ones.<br />
<br />
===Step 1. Identify datafile===<br />
[[File:jsonhowto_fig1.png|thumb|400px|right|Figure 1: File Picker]]<br />
Specify the location of the file.The import page of ARK in 1.1.2 includes a function for importing data directly from a text file or a data stream available online. Currently the file must be available on a web server. (Fig. 1) Enter the location of the file, either a relative path from the ARK root eg data/uploads/your_data.csv or a remote web address eg http://www.example.com/ARK/data?format=json.<br />
Click Submit.<br />
<br />
Depend on the size of the file it may take some time to load into your browser. Once it is loaded it will be available to manipulate very quickly.A spinner will appear while this is happening.<br />
<br />
===Step 2. Determine 'Root' of the data structure.===<br />
<br />
The data you have will likely include information that relates to several ARK items. Within your data structure there will be a level that includes all of these item representations - in the same way as rows in a table. In a csv file where there is a single record per row the root will be 'root>items'. In a more complicated json file (for example one with file level metadata held in the root as well as the items) the records may be contained in an object within the root object. Json allows for extensible representation, so each 'row' in the json file may not include all the same fields. It may be necessary to use the filter tools to only import objects which have a certain criteria.<br />
<br />
The example shown here has many items in the items object in root. The first available column is used to identify each of the contained objects - in this case it is an ARK ID, but it could be anything and may not be unique if it is not unique in your structure. For example if field 1 in your table is a site code or context type.<br />
<br />
[[File:jsonhowto_fig2.png|thumb|400px|left|none|Figure 2: Objects in 'root>items']]<br />
<br />
In order to define this level you will need to navigate to it.The path at the top of the page describes where in the data structure you currently are.The large panel below the advanced options shows the identifiers held in the current level.At the root level of a CSV file the first column will be used as an identifier. Clicking on these will open the data attached to that identifier for viewing.(Fig. 2) It is possible to go up one level using the button at the end of the list of objects, or any part of the location at the top of the page can be used to navigate to that level.When the main window would contain more than 20 items it is truncated and only the first 20 are shown.<br />
<br />
Use the 'this is root' button in the navigation panel at the top of the page when the root is displayed in the main panel.<br />
<br />
===Step 3. Determine ARK IDs===<br />
<br />
[[File:jsonhowto_fig3.png|thumb|right|600px|Figure 3: Root and ARK ID specified]]<br />
<br />
Using the navigation method above find the path to the ARK id within your data. This will often be as simple as 'root>items>item1>Ark_id'.The object that you define the ARK ID in is not important. As long as the objects are structured consistently the ARK IDs will be retrieved automatically based on the root defined in the step above. so for instance the importer will import the ARK ID for METSUR_152 from root > items > METSUR_152 > ARK_id and then the ARK ID for METSUR_153 from root > items > METSUR_153 > ARK_id, and so on.<br />
<br />
<br />
If you do not have ARK IDs in your data refer to the [[#Advanced Options|advanced options]].<br />
Otherwise, click 'This is ARK ID' button below the identifier that contains the ARK ID.<br />
<br />
===Step 4. Test Import===<br />
<br />
[[File:jsonhowto_fig4.png|thumb|right|400px|Figure 4: Example of dry run]]<br />
<br />
Click 'Import this' below the identifier that you would like to import. If you have completed the steps above a new panel will appear below the viewer. Confirm that this looks how you expect – '''this is what will be imported into the database'''.<br />
<br />
The final panel allows you to define which records will be imported. A type must be specified. If you are importing a file it may be necessary to do a separate import of each type. This is because different types have different fields attached to them, the drop down menus respond to the ones above them, presenting only the options available. If they are not completed in order it may result in unexpected behaviour.<br />
<br />
===Step 5.===<br />
When you are happy that you are importing the correct data into the correct field, click 'Submit'.<br />
You will be shown the results of your import. Click 'Import Json' to repeat the process for any other fields you wish to import.<br />
<br />
==Advanced Options==<br />
;Ste_cd:<br />
:The site code that imported items will have, is one is not specified in the data – uses the ARK default if not set. No site code: This check box will remove site codes from the import itemvalue, for use with chains.<br />
;Regular Expression:<br />
:This is used to extract the itemvalue from the ARK ID column – by default it grabs the first run of numbers in the field.<br />
;Start at Arbitrary Number:<br />
:If you wish to create a sequence of numbers starting from a given point, (including 1) you will need to specify this here, and the number to start on. A unique 'ARK ID' must still be specified in the data, but it will not be used.</div>Mikehttps://ark.lparchaeology.com/wiki/index.php?title=Basic_Installation&diff=124883Basic Installation2018-03-19T14:09:55Z<p>Mike: </p>
<hr />
<div>If you are comfortable setting up an Apache webserver and a MySQL database, then setting up ARK is very easy. Follow the guide below to get your ARK instance installed and set up.<br />
<br />
If you are unfamiliar with the setup and administration of webservers along with the security risks to your system and data this entails, please consider contacting the ARK team to acquire a fully running ARK.<br />
<br />
There are some further resources for learning about ARK and how to install it and some of the required dependencies [http://ark.lparchaeology.com/resources/ on the ARK website].<br />
<br />
===Download source===<br />
<br />
Download the latest source code from http://sourceforge.net/projects/arkdb/ sourceforge.<br />
<br />
===Check and Install Dependencies===<br />
<br />
====Dependencies====<br />
<br />
ARK requires the following open source software packages to be installed on your server:<br />
<br />
#Apache<br />
#PHP > 5.4 and < 7.0<br />
#MySQL<br />
<br />
ARK is able to connect to any WMC/WFS compliant server of geographic information. Mapserver is no longer an ARK dependency. This means that that administrators can connect to a geographic database of their choice, to suit their needs. Details on [http://mapserver.org/ Mapserver] setup can be found on their website.<br />
<br />
ARK 1.1 requires PHP 5.4 or later. PHP 5.5 will run ARK but will produce "deprecated" errors as some functions used in ARK have been deprecated as of 5.5. If you are using PHP 5.5 and see deprecated messages, these can be hidden by following the relevant PHP documentation. These functions will be updated in a future release of ARK. ARK 1.X will not run on PHP 7.0 or later. ARK 2 is currently in development, which will run on PHP versions greater than 7.0.<br />
<br />
ARK runs on top of these packages and does not provide any set up of them or additional security to them. If you are using your server exposed to the internet, it is essential that you follow good standard security practice for the configuration of your packages. This setup is outside of the scope of this documentation and you are advised to undertake your research carefully.<br />
<br />
In addition the following packages are useful:<br />
<br />
#phpMyAdmin<br />
<br />
Once you have Apache and MySQL running, phpMyAdmin is a very convenient tool for administering databases. More information on how to use phpMyadmin can be found on their [http://docs.phpmyadmin.net/en/latest/user.html website].<br />
<br />
====on LINUX====<br />
<br />
On most Linux distributions these packages will be installed by default.<br />
<br />
=====Apache=====<br />
<br />
Most Linux distributions include an Apache webserver. Consult the documentation on your installation for instructions on how to configure your server. If your service path is configured, the service can be started with:<br />
<br />
<code> sudo service httpd start </code><br />
<br />
If you need to install Apache, instructions can be found on their [http://httpd.apache.org/docs/current/install.html website].<br />
<br />
=====PHP=====<br />
<br />
Most Linux distributions include PHP. Make sure that the version number of PHP installed on your system matches the requirements of ARK.<br />
<br />
If you need to install PHP, instructions can be found on their [http://www.php.net/manual/en/install.php website].<br />
<br />
=====MySQL=====<br />
<br />
MySQL is included in many Linux distributions, but if you do not have it you may need to install <code>mysql-server</code> and <code>mysql</code>, instructions are available on the [http://dev.mysql.com/doc/refman/5.1/en/binary-installation.html MySQL website]. <br />
<br />
You will need to secure the installation, <code>sudo /usr/bin/mysql_secure_installation</code>.<br />
<br />
====on OSX or WINDOWS====<br />
<br />
There are several ways to manually install each of the required packages on OSX or Windows. These methods will work perfectly well with ARK and if you are comfortable installing and configuring the required packages, please go ahead and do so. For OSX the instructions will be similar to the Linux instructions.<br />
<br />
If you would prefer a simple package-based installer, please consider using either WAMP/MAMP or if you are using Windows, MS4W also provides a good way to install a complete WAMP stack along with a Mapserver instance.<br />
<br />
=====MAMP/WAMP=====<br />
<br />
For Mac OSX or Windows, the required packages are available as a bundle known as [http://www.mamp.info/en/index.php MAMP] for Mac or [http://www.wampserver.com/en/ WAMP] Windows. Their websites detail the installation procedures for these services.<br />
<br />
=====MS4W=====<br />
<br />
This package provides a standalone WAMP stack with a Mapserver instance. We have provided some walk through guidance [http://ark.lparchaeology.com/installing-mapserver-and-ms4w/ on the ARK website] for installing this package.<br />
<br />
==Unpack ARK==<br />
<br />
The ARK download consists of a folder of containing php code and several 'config' folders - the config folders contain the SQL code for that configuration. The first step is to copy the arkv1_1 folder into the desired location in your web server directory. Consult the documentation of your distribution if you are unsure where this is. It is likely to be something like <code>/var/www/</code> or <code>/srv/www/</code>. You will then need to either select one of the preconfigured configs and rename it to 'config' or create your own.<br />
<br />
You can now rename the ARK package's folder to your desired name. If you do so, make a note of the folder name because you will need to update the settings to match it. This folder name will be used to navigate your ARK using your browser (in the URL), so call it something short, easy to remember and unique to your installation.<br />
<br />
==Create Database and User==<br />
<br />
You now need to create a new database, import the SQL into it and create a new user on your MySQL server.<br />
<br />
The basic ARK database is included in the ARK package as an SQL file. <br />
<br />
ARK only requires one user on the database, this is the user that the php code will use to access the MySQL database. Actual end users of your ARK will be created later using the tools provided by ARK.<br />
<br />
The easiest way to accomplish these steps is to use phpMyAdmin, but you can also accomplish it using the command line mysql tools:<br />
<br />
===Using phpMyAdmin===<br />
<br />
Use the phpMyAdmin interface to create the new database. This is the database that will contain your ARK data, so make a note of the name so that you can update the settings files to match.<br />
<br />
Now create a new user and grant them privileges on the new database.<br />
<br />
Import the arkv1_1.sql file from the ARK source code into your database. Select the newly created database and then go to the Import tab and Browse to the .zip file that contains the SQL statement. This will copy all of the relevant tables to the database.<br />
<br />
===Using the Command Line===<br />
<br />
You can use MySQL from the command line to add the 'arkuser' user to the MySQL installation for instance: <br />
<code>CREATE USER arkuser'@'%' IDENTIFIED BY 'XXXXXX';<br />
GRANT ALL PRIVILEGES ON *.* TO 'arkuser'@'%' WITH GRANT OPTION;</code><br />
<br />
The values in this must be changed in order to secure your database. See the [http://dev.mysql.com/doc/ mysql manual] for more information. <br />
<br />
You will also need to create a database for your ARK to connect to. <br />
<code>CREATE DATABASE arkv1_1</code><br />
<br />
You are encouraged to change these default names see the [http://dev.mysql.com/doc/ mysql manual] for more information.<br />
<br />
You should now load the ARK SQL into your database by using the <code>Source</code> command on the mysql terminal specifying the location of the arkv1_1.sql file.<br />
<br />
===Use install_ark===<br />
With ARK1.2 there is a bash script which will follow the steps above. This is the quickest way to set up an ARK, but you may wish to change the username or password from the suggested defaults this script will use.<br />
<br />
==Initial Setup==<br />
<br />
===Config===<br />
<br />
You will need to edit some of the configuration directives in the settings files located inside the <code>config</code> folder of your ARK installation.<br />
<br />
====settings====<br />
<br />
Start with the general <code>settings.php</code> file.<br />
<br />
You must enter the path of the ark relative to the root of the server. You can also update the name of your ARK, and the markup nickname (nname) for a piece of markup containing the human readable name of your ARK. This allows you to customise the name of your ARK as it appears in page headers and titles. You will now need to add the markup to the ARK database. <br />
<br />
You should also set your default site code in this file. Make a note of your site code(s) and be sure to add them to the ARK database using the form provided in the data entry page as soon as you log in. You will now need to add the site code to the ARK database.<br />
<br />
====environment settings====<br />
<br />
IIf you ahve created a database yourself, rather than using the install_ark script you will need to edit your <code>env_settings.php</code> file. This file should be self-explanatory, however there is an explanation of the relevant settings [http://ark.lparchaeology.com/wiki/index.php/Env_settings.php on this wiki page].<br />
<br />
Change the <code>$server</code> directive on line 52 to match your server type, either mac, windows or linux. <br />
<br />
Then in the correct code block of your server type, alter the paths to match the locations of the relevant directories in your ARK environment.<br />
<br />
Make sure that the various path directives are correct. Be aware that you should update these to match your new ark folder name as set up earlier. For example:<br />
<br />
<pre><br />
// The folder name of THIS instance of ARK (relative to the domain)<br />
$ark_dir<br />
// The server path to the ark directory<br />
$ark_server_path<br />
// The path to the PEAR installation<br />
$pear_path<br />
</pre><br />
<br />
Both local and remote paths for OpenLayers are offered, use // to comment out the line that you are not using.<br />
<br />
Alter the database settings at the end of the file to match the database and user that you created earlier.<br />
<br />
<b>You are now ready to connect to your ARK!</b><br />
<br />
===Preflight Checks===<br />
<br />
Point your browser to the preflight checks file included in ARK to test that you have configured ARK correctly. It should be available on your server at www.example.com/'your ark directory'/config/preflight_checks.php. Depending on your setup you may be able to access it at localhost. Check your server documentation for details.<br />
<br />
This tool will list set up options with either PASS or FAIL. If your config FAILs please follow the instructions on the preflight checks page, or refer to other pages on the wiki for more details.<br />
<br />
====File Uploads and phMagick====<br />
<br />
It is highly likely that a standard ARK install will fail on some or all of the tests in this section of the pre-flight tests.<br />
<br />
If you intend to upload files such as PDFs or images to your ARK and would like them to have thumbnails, then you will need to pass all of these tests. If not, then ignore this section of the tests.<br />
<br />
For further information on how to install the required PHP extensions, imageMagick and phMagick, please consult the relevant online documentation.<br />
<br />
===Browse to your ARK===<br />
<br />
Point your browser to the newly created ARK directory, typically this would be <code>www.example.com/ark_dir/</code> or <code>localhost/ark_dir/</code> although the exact URL will vary according to both your hostname and your ARK directory settings.<br />
<br />
After browsing to the ARK directory you should see the login pages for the new ARK system. The default ARK is created with the following admin user account:<br />
<br />
<pre><br />
username: doe_jd<br />
password: janedoe<br />
</pre><br />
<br />
Log in using these details and then finalise your configuration.<br />
<br />
===Final Configuration===<br />
<br />
If you successfully logged in as Jane Doe, pat yourself on the back and now mask sure you finish the last few steps:<br />
<br />
You now need to [http://ark.lparchaeology.com/wiki/index.php/User_Administration create a new user] for yourself on ARK and make yourself and administrator. Once you have created this new administrator, you should logout and log back in again using you new username.<br />
<br />
You <b>should</b> now remove the default doe_jd user to prevent unauthorised access to your system.<br />
<br />
Now [http://ark.lparchaeology.com/wiki/index.php/Markup_Administration add markup] for the ARK name you specified in the settings file.<br />
<br />
Now [http://ark.lparchaeology.com/wiki/index.php/site_code_administration add your default site code] as specified in the settings file.<br />
<br />
==Debug==<br />
<br />
If you experience problems after following the steps above, there are a range of possible reasons that your local setup may not be correct. In order to display debug messages, go to your settings file and turn on debug by adjusting the settings from:<br />
<br />
<pre><br />
error_reporting(0); // Turn off all error reporting - USE THIS FOR PRODUCTION SITES<br />
//error_reporting(E_ALL); // Report all PHP errors (see changelog) - USE THIS FOR SETUP AND TESTING<br />
</pre><br />
<br />
to<br />
<br />
<pre><br />
//error_reporting(0); // Turn off all error reporting - USE THIS FOR PRODUCTION SITES<br />
error_reporting(E_ALL); // Report all PHP errors (see changelog) - USE THIS FOR SETUP AND TESTING<br />
</pre><br />
<br />
==Cleanup==<br />
<br />
Once you have passed the pre-flight checks you should remove (delete or make inaccessible) this file so that it cannot be seen by unwanted users.<br />
<br />
[[category:Administrator]]</div>Mikehttps://ark.lparchaeology.com/wiki/index.php?title=Basic_Installation&diff=124882Basic Installation2018-03-19T14:05:09Z<p>Mike: /* environment settings */</p>
<hr />
<div>If you are comfortable setting up an Apache webserver and a MySQL database, then setting up ARK is very easy. Follow the guide below to get your ARK instance installed and set up.<br />
<br />
If you are unfamiliar with the setup and administration of webservers along with the security risks to your system and data this entails, please consider contacting the ARK team to acquire a fully running ARK.<br />
<br />
There are some further resources for learning about ARK and how to install it and some of the required dependencies [http://ark.lparchaeology.com/resources/ on the ARK website].<br />
<br />
===Download source===<br />
<br />
Download the latest source code from http://sourceforge.net/projects/arkdb/ sourceforge.<br />
<br />
===Check and Install Dependencies===<br />
<br />
====Dependencies====<br />
<br />
ARK requires the following open source software packages to be installed on your server:<br />
<br />
#Apache<br />
#PHP > 5.4 and < 7.0<br />
#MySQL<br />
<br />
ARK is able to connect to any WMC/WFS compliant server of geographic information. Mapserver is no longer an ARK dependency. This means that that administrators can connect to a geographic database of their choice, to suit their needs. Details on [http://mapserver.org/ Mapserver] setup can be found on their website.<br />
<br />
ARK 1.1 requires PHP 5.4 or later. PHP 5.5 will run ARK but will produce "deprecated" errors as some functions used in ARK have been deprecated as of 5.5. If you are using PHP 5.5 and see deprecated messages, these can be hidden by following the relevant PHP documentation. These functions will be updated in a future release of ARK. ARK 1.X will not run on PHP 7.0 or later. ARK 2 is currently in development, which will run on PHP versions greater than 7.0.<br />
<br />
ARK runs on top of these packages and does not provide any set up of them or additional security to them. If you are using your server exposed to the internet, it is essential that you follow good standard security practice for the configuration of your packages. This setup is outside of the scope of this documentation and you are advised to undertake your research carefully.<br />
<br />
In addition the following packages are useful:<br />
<br />
#phpMyAdmin<br />
<br />
Once you have Apache and MySQL running, phpMyAdmin is a very convenient tool for administering databases. More information on how to use phpMyadmin can be found on their [http://docs.phpmyadmin.net/en/latest/user.html website].<br />
<br />
====on LINUX====<br />
<br />
On most Linux distributions these packages will be installed by default.<br />
<br />
=====Apache=====<br />
<br />
Most Linux distributions include an Apache webserver. Consult the documentation on your installation for instructions on how to configure your server. If your service path is configured, the service can be started with:<br />
<br />
<code> sudo service httpd start </code><br />
<br />
If you need to install Apache, instructions can be found on their [http://httpd.apache.org/docs/current/install.html website].<br />
<br />
=====PHP=====<br />
<br />
Most Linux distributions include PHP. Make sure that the version number of PHP installed on your system matches the requirements of ARK.<br />
<br />
If you need to install PHP, instructions can be found on their [http://www.php.net/manual/en/install.php website].<br />
<br />
=====MySQL=====<br />
<br />
MySQL is included in many Linux distributions, but if you do not have it you may need to install <code>mysql-server</code> and <code>mysql</code>, instructions are available on the [http://dev.mysql.com/doc/refman/5.1/en/binary-installation.html MySQL website]. <br />
<br />
You will need to secure the installation, <code>sudo /usr/bin/mysql_secure_installation</code>.<br />
<br />
====on OSX or WINDOWS====<br />
<br />
There are several ways to manually install each of the required packages on OSX or Windows. These methods will work perfectly well with ARK and if you are comfortable installing and configuring the required packages, please go ahead and do so. For OSX the instructions will be similar to the Linux instructions.<br />
<br />
If you would prefer a simple package-based installer, please consider using either WAMP/MAMP or if you are using Windows, MS4W also provides a good way to install a complete WAMP stack along with a Mapserver instance.<br />
<br />
=====MAMP/WAMP=====<br />
<br />
For Mac OSX or Windows, the required packages are available as a bundle known as [http://www.mamp.info/en/index.php MAMP] for Mac or [http://www.wampserver.com/en/ WAMP] Windows. Their websites detail the installation procedures for these services.<br />
<br />
=====MS4W=====<br />
<br />
This package provides a standalone WAMP stack with a Mapserver instance. We have provided some walk through guidance [http://ark.lparchaeology.com/installing-mapserver-and-ms4w/ on the ARK website] for installing this package.<br />
<br />
==Unpack ARK==<br />
<br />
The ARK download consists of a folder of containing php code and several 'config' folders - the config folders contain the SQL code for that configuration. The first step is to copy the arkv1_1 folder into the desired location in your web server directory. Consult the documentation of your distribution if you are unsure where this is. It is likely to be something like <code>/var/www/</code> or <code>/srv/www/</code>. You will then need to either select one of the preconfigured configs and rename it to 'config' or create your own.<br />
<br />
You can now rename the ARK package's folder to your desired name. If you do so, make a note of the folder name because you will need to update the settings to match it. This folder name will be used to navigate your ARK using your browser (in the URL), so call it something short, easy to remember and unique to your installation.<br />
<br />
==Create Database and User==<br />
<br />
You now need to create a new database, import the SQL into it and create a new user on your MySQL server.<br />
<br />
The basic ARK database is included in the ARK package as an SQL file. <br />
<br />
ARK only requires one user on the database, this is the user that the php code will use to access the MySQL database. Actual end users of your ARK will be created later using the tools provided by ARK.<br />
<br />
The easiest way to accomplish these steps is to use phpMyAdmin, but you can also accomplish it using the command line mysql tools:<br />
<br />
===Using phpMyAdmin===<br />
<br />
Use the phpMyAdmin interface to create the new database. This is the database that will contain your ARK data, so make a note of the name so that you can update the settings files to match.<br />
<br />
Now create a new user and grant them privileges on the new database.<br />
<br />
Import the arkv1_1.sql file from the ARK source code into your database. Select the newly created database and then go to the Import tab and Browse to the .zip file that contains the SQL statement. This will copy all of the relevant tables to the database.<br />
<br />
===Using the Command Line===<br />
<br />
You can use MySQL from the command line to add the 'arkuser' user to the MySQL installation for instance: <br />
<code>CREATE USER arkuser'@'%' IDENTIFIED BY 'XXXXXX';<br />
GRANT ALL PRIVILEGES ON *.* TO 'arkuser'@'%' WITH GRANT OPTION;</code><br />
<br />
The values in this must be changed in order to secure your database. See the [http://dev.mysql.com/doc/ mysql manual] for more information. <br />
<br />
You will also need to create a database for your ARK to connect to. <br />
<code>CREATE DATABASE arkv1_1</code><br />
<br />
You are encouraged to change these default names see the [http://dev.mysql.com/doc/ mysql manual] for more information.<br />
<br />
You should now load the ARK SQL into your database by using the <code>Source</code> command on the mysql terminal specifying the location of the arkv1_1.sql file.<br />
<br />
===Use install_ark===<br />
With ARK1.2 there is a bash script which will follow the steps above. This is the quickest way to set up an ARK, but you may wish to change the username or password from the suggested defaults this script will use.<br />
<br />
==Initial Setup==<br />
<br />
===Config===<br />
<br />
You will need to edit some of the configuration directives in the settings files located inside the <code>config</code> folder of your ARK installation.<br />
<br />
====settings====<br />
<br />
Start with the general <code>settings.php</code> file.<br />
<br />
You must enter the path of the ark relative to the root of the server. You can also update the name of your ARK, and the markup nickname (nname) for a piece of markup containing the human readable name of your ARK. This allows you to customise the name of your ARK as it appears in page headers and titles. You will now need to add the markup to the ARK database. <br />
<br />
You should also set your default site code in this file. Make a note of your site code(s) and be sure to add them to the ARK database using the form provided in the data entry page as soon as you log in. You will now need to add the site code to the ARK database.<br />
<br />
====environment settings====<br />
<br />
IIf you ahve created a database yourself, rather than using the install_ark script you will need to edit your <code>env_settings.php</code> file. This file should be self-explanatory, however there is an explanation of the relevant settings [http://ark.lparchaeology.com/wiki/index.php/Env_settings.php on this wiki page].<br />
<br />
Change the <code>$server</code> directive on line 52 to match your server type, either mac, windows or linux. <br />
<br />
Then in the correct code block of your server type, alter the paths to match the locations of the relevant directories in your ARK environment.<br />
<br />
Make sure that the various path directives are correct. Be aware that you should update these to match your new ark folder name as set up earlier. For example:<br />
<br />
<pre><br />
// The folder name of THIS instance of ARK (relative to the domain)<br />
$ark_dir<br />
// The server path to the ark directory<br />
$ark_server_path<br />
// The path to the PEAR installation<br />
$pear_path<br />
</pre><br />
<br />
Both local and remote paths for OpenLayers are offered, use // to comment out the line that you are not using.<br />
<br />
Alter the database settings at the end of the file to match the database and user that you created earlier.<br />
<br />
<b>You are now ready to connect to your ARK!</b><br />
<br />
===Preflight Checks===<br />
<br />
Point your browser to the preflight checks file included in ARK to test that you have configured ARK correctly. It should be available on your server at www.example.com/'your ark directory'/config/preflight_checks.php. Depending on your setup you may be able to access it at localhost. Check your server documentation for details.<br />
<br />
This tool will list set up options with either PASS or FAIL. If your config FAILs please follow the instructions on the preflight checks page, or refer to other pages on the wiki for more details.<br />
<br />
====File Uploads and phMagick====<br />
<br />
It is highly likely that a standard ARK install will fail on some or all of the tests in this section of the pre-flight tests.<br />
<br />
If you intend to upload files such as PDFs or images to your ARK and would like them to have thumbnails, then you will need to pass all of these tests. If not, then ignore this section of the tests.<br />
<br />
For further information on how to install the required PHP extensions, imageMagick and phMagick, please consult the relevant online documentation.<br />
<br />
===Browse to your ARK===<br />
<br />
Point your browser to the newly created ARK directory, typically this would be <code>www.example.com/ark_dir/</code> or <code>localhost/ark_dir/</code> although the exact URL will vary according to both your hostname and your ARK directory settings.<br />
<br />
After browsing to the ARK directory you should see the login pages for the new ARK system. The default ARK is created with the following admin user account:<br />
<br />
<pre><br />
username: doe_jd<br />
password: janedoe<br />
</pre><br />
<br />
Log in using these details and then finalise your configuration.<br />
<br />
===Final Configuration===<br />
<br />
If you successfully logged in as Jane Doe, pat yourself on the back and now mask sure you finish the last few steps:<br />
<br />
You now need to [http://ark.lparchaeology.com/wiki/index.php/User_Administration create a new user] for yourself on ARK and make yourself and administrator. Once you have created this new administrator, you should logout and log back in again using you new username.<br />
<br />
You <b>should</b> now remove the default doe_jd user to prevent unauthorised access to your system.<br />
<br />
Now [http://ark.lparchaeology.com/wiki/index.php/Markup_Administration add markup] for the ARK name you specified in the settings file.<br />
<br />
Now [http://ark.lparchaeology.com/wiki/index.php/site_code_administration add your default site code] as specified in the settings file.<br />
<br />
==Debug==<br />
<br />
If you experience problems after following the steps above, there are a range of possible reasons that your local setup may not be correct. In order to display debug messages, go to your settings file and turn on debug by adjusting the settings from:<br />
<br />
<pre><br />
error_reporting(0); // Turn off all error reporting - USE THIS FOR PRODUCTION SITES<br />
//error_reporting(E_ALL); // Report all PHP errors (see changelog) - USE THIS FOR SETUP AND TESTING<br />
</pre><br />
<br />
to<br />
<br />
<pre><br />
//error_reporting(0); // Turn off all error reporting - USE THIS FOR PRODUCTION SITES<br />
error_reporting(E_ALL); // Report all PHP errors (see changelog) - USE THIS FOR SETUP AND TESTING<br />
</pre><br />
<br />
==Cleanup==<br />
<br />
Once you have passed the pre-flight checks you should remove (delete or make inaccessible) this file so that it cannot be seen by unwanted users.<br />
<br />
Remove the Jane Doe default user. This is to prevent unauthorised access to your ARK via this account!<br />
<br />
If you plan on using your ARK on an unsecured network (such as the Internet) you should consider how you intend to encrypt traffic between the users and the server. You should consider how you will be securing passwords and login credentials against various kind of attacks. Such security measures are important but are outside the scope of this install document.<br />
<br />
[[category:Administrator]]</div>Mikehttps://ark.lparchaeology.com/wiki/index.php?title=Basic_Installation&diff=124881Basic Installation2018-03-19T14:04:10Z<p>Mike: /* Initial Setup */</p>
<hr />
<div>If you are comfortable setting up an Apache webserver and a MySQL database, then setting up ARK is very easy. Follow the guide below to get your ARK instance installed and set up.<br />
<br />
If you are unfamiliar with the setup and administration of webservers along with the security risks to your system and data this entails, please consider contacting the ARK team to acquire a fully running ARK.<br />
<br />
There are some further resources for learning about ARK and how to install it and some of the required dependencies [http://ark.lparchaeology.com/resources/ on the ARK website].<br />
<br />
===Download source===<br />
<br />
Download the latest source code from http://sourceforge.net/projects/arkdb/ sourceforge.<br />
<br />
===Check and Install Dependencies===<br />
<br />
====Dependencies====<br />
<br />
ARK requires the following open source software packages to be installed on your server:<br />
<br />
#Apache<br />
#PHP > 5.4 and < 7.0<br />
#MySQL<br />
<br />
ARK is able to connect to any WMC/WFS compliant server of geographic information. Mapserver is no longer an ARK dependency. This means that that administrators can connect to a geographic database of their choice, to suit their needs. Details on [http://mapserver.org/ Mapserver] setup can be found on their website.<br />
<br />
ARK 1.1 requires PHP 5.4 or later. PHP 5.5 will run ARK but will produce "deprecated" errors as some functions used in ARK have been deprecated as of 5.5. If you are using PHP 5.5 and see deprecated messages, these can be hidden by following the relevant PHP documentation. These functions will be updated in a future release of ARK. ARK 1.X will not run on PHP 7.0 or later. ARK 2 is currently in development, which will run on PHP versions greater than 7.0.<br />
<br />
ARK runs on top of these packages and does not provide any set up of them or additional security to them. If you are using your server exposed to the internet, it is essential that you follow good standard security practice for the configuration of your packages. This setup is outside of the scope of this documentation and you are advised to undertake your research carefully.<br />
<br />
In addition the following packages are useful:<br />
<br />
#phpMyAdmin<br />
<br />
Once you have Apache and MySQL running, phpMyAdmin is a very convenient tool for administering databases. More information on how to use phpMyadmin can be found on their [http://docs.phpmyadmin.net/en/latest/user.html website].<br />
<br />
====on LINUX====<br />
<br />
On most Linux distributions these packages will be installed by default.<br />
<br />
=====Apache=====<br />
<br />
Most Linux distributions include an Apache webserver. Consult the documentation on your installation for instructions on how to configure your server. If your service path is configured, the service can be started with:<br />
<br />
<code> sudo service httpd start </code><br />
<br />
If you need to install Apache, instructions can be found on their [http://httpd.apache.org/docs/current/install.html website].<br />
<br />
=====PHP=====<br />
<br />
Most Linux distributions include PHP. Make sure that the version number of PHP installed on your system matches the requirements of ARK.<br />
<br />
If you need to install PHP, instructions can be found on their [http://www.php.net/manual/en/install.php website].<br />
<br />
=====MySQL=====<br />
<br />
MySQL is included in many Linux distributions, but if you do not have it you may need to install <code>mysql-server</code> and <code>mysql</code>, instructions are available on the [http://dev.mysql.com/doc/refman/5.1/en/binary-installation.html MySQL website]. <br />
<br />
You will need to secure the installation, <code>sudo /usr/bin/mysql_secure_installation</code>.<br />
<br />
====on OSX or WINDOWS====<br />
<br />
There are several ways to manually install each of the required packages on OSX or Windows. These methods will work perfectly well with ARK and if you are comfortable installing and configuring the required packages, please go ahead and do so. For OSX the instructions will be similar to the Linux instructions.<br />
<br />
If you would prefer a simple package-based installer, please consider using either WAMP/MAMP or if you are using Windows, MS4W also provides a good way to install a complete WAMP stack along with a Mapserver instance.<br />
<br />
=====MAMP/WAMP=====<br />
<br />
For Mac OSX or Windows, the required packages are available as a bundle known as [http://www.mamp.info/en/index.php MAMP] for Mac or [http://www.wampserver.com/en/ WAMP] Windows. Their websites detail the installation procedures for these services.<br />
<br />
=====MS4W=====<br />
<br />
This package provides a standalone WAMP stack with a Mapserver instance. We have provided some walk through guidance [http://ark.lparchaeology.com/installing-mapserver-and-ms4w/ on the ARK website] for installing this package.<br />
<br />
==Unpack ARK==<br />
<br />
The ARK download consists of a folder of containing php code and several 'config' folders - the config folders contain the SQL code for that configuration. The first step is to copy the arkv1_1 folder into the desired location in your web server directory. Consult the documentation of your distribution if you are unsure where this is. It is likely to be something like <code>/var/www/</code> or <code>/srv/www/</code>. You will then need to either select one of the preconfigured configs and rename it to 'config' or create your own.<br />
<br />
You can now rename the ARK package's folder to your desired name. If you do so, make a note of the folder name because you will need to update the settings to match it. This folder name will be used to navigate your ARK using your browser (in the URL), so call it something short, easy to remember and unique to your installation.<br />
<br />
==Create Database and User==<br />
<br />
You now need to create a new database, import the SQL into it and create a new user on your MySQL server.<br />
<br />
The basic ARK database is included in the ARK package as an SQL file. <br />
<br />
ARK only requires one user on the database, this is the user that the php code will use to access the MySQL database. Actual end users of your ARK will be created later using the tools provided by ARK.<br />
<br />
The easiest way to accomplish these steps is to use phpMyAdmin, but you can also accomplish it using the command line mysql tools:<br />
<br />
===Using phpMyAdmin===<br />
<br />
Use the phpMyAdmin interface to create the new database. This is the database that will contain your ARK data, so make a note of the name so that you can update the settings files to match.<br />
<br />
Now create a new user and grant them privileges on the new database.<br />
<br />
Import the arkv1_1.sql file from the ARK source code into your database. Select the newly created database and then go to the Import tab and Browse to the .zip file that contains the SQL statement. This will copy all of the relevant tables to the database.<br />
<br />
===Using the Command Line===<br />
<br />
You can use MySQL from the command line to add the 'arkuser' user to the MySQL installation for instance: <br />
<code>CREATE USER arkuser'@'%' IDENTIFIED BY 'XXXXXX';<br />
GRANT ALL PRIVILEGES ON *.* TO 'arkuser'@'%' WITH GRANT OPTION;</code><br />
<br />
The values in this must be changed in order to secure your database. See the [http://dev.mysql.com/doc/ mysql manual] for more information. <br />
<br />
You will also need to create a database for your ARK to connect to. <br />
<code>CREATE DATABASE arkv1_1</code><br />
<br />
You are encouraged to change these default names see the [http://dev.mysql.com/doc/ mysql manual] for more information.<br />
<br />
You should now load the ARK SQL into your database by using the <code>Source</code> command on the mysql terminal specifying the location of the arkv1_1.sql file.<br />
<br />
===Use install_ark===<br />
With ARK1.2 there is a bash script which will follow the steps above. This is the quickest way to set up an ARK, but you may wish to change the username or password from the suggested defaults this script will use.<br />
<br />
==Initial Setup==<br />
<br />
===Config===<br />
<br />
You will need to edit some of the configuration directives in the settings files located inside the <code>config</code> folder of your ARK installation.<br />
<br />
====settings====<br />
<br />
Start with the general <code>settings.php</code> file.<br />
<br />
You must enter the path of the ark relative to the root of the server. You can also update the name of your ARK, and the markup nickname (nname) for a piece of markup containing the human readable name of your ARK. This allows you to customise the name of your ARK as it appears in page headers and titles. You will now need to add the markup to the ARK database. <br />
<br />
You should also set your default site code in this file. Make a note of your site code(s) and be sure to add them to the ARK database using the form provided in the data entry page as soon as you log in. You will now need to add the site code to the ARK database.<br />
<br />
====environment settings====<br />
<br />
Next edit your <code>env_settings.php</code> file. This file should be self-explanatory, however there is an explanation of the relevant settings [http://ark.lparchaeology.com/wiki/index.php/Env_settings.php on this wiki page].<br />
<br />
Change the <code>$server</code> directive on line 52 to match your server type, either mac, windows or linux. <br />
<br />
Then in the correct code block of your server type, alter the paths to match the locations of the relevant directories in your ARK environment.<br />
<br />
Make sure that the various path directives are correct. Be aware that you should update these to match your new ark folder name as set up earlier. For example:<br />
<br />
<pre><br />
// The folder name of THIS instance of ARK (relative to the domain)<br />
$ark_dir<br />
// The server path to the ark directory<br />
$ark_server_path<br />
// The path to the PEAR installation<br />
$pear_path<br />
</pre><br />
<br />
Both local and remote paths for OpenLayers are offered, use // to comment out the line that you are not using.<br />
<br />
Alter the database settings at the end of the file to match the database and user that you created earlier.<br />
<br />
<b>You are now ready to connect to your ARK!</b><br />
<br />
===Preflight Checks===<br />
<br />
Point your browser to the preflight checks file included in ARK to test that you have configured ARK correctly. It should be available on your server at www.example.com/'your ark directory'/config/preflight_checks.php. Depending on your setup you may be able to access it at localhost. Check your server documentation for details.<br />
<br />
This tool will list set up options with either PASS or FAIL. If your config FAILs please follow the instructions on the preflight checks page, or refer to other pages on the wiki for more details.<br />
<br />
====File Uploads and phMagick====<br />
<br />
It is highly likely that a standard ARK install will fail on some or all of the tests in this section of the pre-flight tests.<br />
<br />
If you intend to upload files such as PDFs or images to your ARK and would like them to have thumbnails, then you will need to pass all of these tests. If not, then ignore this section of the tests.<br />
<br />
For further information on how to install the required PHP extensions, imageMagick and phMagick, please consult the relevant online documentation.<br />
<br />
===Browse to your ARK===<br />
<br />
Point your browser to the newly created ARK directory, typically this would be <code>www.example.com/ark_dir/</code> or <code>localhost/ark_dir/</code> although the exact URL will vary according to both your hostname and your ARK directory settings.<br />
<br />
After browsing to the ARK directory you should see the login pages for the new ARK system. The default ARK is created with the following admin user account:<br />
<br />
<pre><br />
username: doe_jd<br />
password: janedoe<br />
</pre><br />
<br />
Log in using these details and then finalise your configuration.<br />
<br />
===Final Configuration===<br />
<br />
If you successfully logged in as Jane Doe, pat yourself on the back and now mask sure you finish the last few steps:<br />
<br />
You now need to [http://ark.lparchaeology.com/wiki/index.php/User_Administration create a new user] for yourself on ARK and make yourself and administrator. Once you have created this new administrator, you should logout and log back in again using you new username.<br />
<br />
You <b>should</b> now remove the default doe_jd user to prevent unauthorised access to your system.<br />
<br />
Now [http://ark.lparchaeology.com/wiki/index.php/Markup_Administration add markup] for the ARK name you specified in the settings file.<br />
<br />
Now [http://ark.lparchaeology.com/wiki/index.php/site_code_administration add your default site code] as specified in the settings file.<br />
<br />
==Debug==<br />
<br />
If you experience problems after following the steps above, there are a range of possible reasons that your local setup may not be correct. In order to display debug messages, go to your settings file and turn on debug by adjusting the settings from:<br />
<br />
<pre><br />
error_reporting(0); // Turn off all error reporting - USE THIS FOR PRODUCTION SITES<br />
//error_reporting(E_ALL); // Report all PHP errors (see changelog) - USE THIS FOR SETUP AND TESTING<br />
</pre><br />
<br />
to<br />
<br />
<pre><br />
//error_reporting(0); // Turn off all error reporting - USE THIS FOR PRODUCTION SITES<br />
error_reporting(E_ALL); // Report all PHP errors (see changelog) - USE THIS FOR SETUP AND TESTING<br />
</pre><br />
<br />
==Cleanup==<br />
<br />
Once you have passed the pre-flight checks you should remove (delete or make inaccessible) this file so that it cannot be seen by unwanted users.<br />
<br />
Remove the Jane Doe default user. This is to prevent unauthorised access to your ARK via this account!<br />
<br />
If you plan on using your ARK on an unsecured network (such as the Internet) you should consider how you intend to encrypt traffic between the users and the server. You should consider how you will be securing passwords and login credentials against various kind of attacks. Such security measures are important but are outside the scope of this install document.<br />
<br />
[[category:Administrator]]</div>Mikehttps://ark.lparchaeology.com/wiki/index.php?title=Basic_Installation&diff=124880Basic Installation2018-03-19T12:34:08Z<p>Mike: /* Create Database and User */</p>
<hr />
<div>If you are comfortable setting up an Apache webserver and a MySQL database, then setting up ARK is very easy. Follow the guide below to get your ARK instance installed and set up.<br />
<br />
If you are unfamiliar with the setup and administration of webservers along with the security risks to your system and data this entails, please consider contacting the ARK team to acquire a fully running ARK.<br />
<br />
There are some further resources for learning about ARK and how to install it and some of the required dependencies [http://ark.lparchaeology.com/resources/ on the ARK website].<br />
<br />
===Download source===<br />
<br />
Download the latest source code from http://sourceforge.net/projects/arkdb/ sourceforge.<br />
<br />
===Check and Install Dependencies===<br />
<br />
====Dependencies====<br />
<br />
ARK requires the following open source software packages to be installed on your server:<br />
<br />
#Apache<br />
#PHP > 5.4 and < 7.0<br />
#MySQL<br />
<br />
ARK is able to connect to any WMC/WFS compliant server of geographic information. Mapserver is no longer an ARK dependency. This means that that administrators can connect to a geographic database of their choice, to suit their needs. Details on [http://mapserver.org/ Mapserver] setup can be found on their website.<br />
<br />
ARK 1.1 requires PHP 5.4 or later. PHP 5.5 will run ARK but will produce "deprecated" errors as some functions used in ARK have been deprecated as of 5.5. If you are using PHP 5.5 and see deprecated messages, these can be hidden by following the relevant PHP documentation. These functions will be updated in a future release of ARK. ARK 1.X will not run on PHP 7.0 or later. ARK 2 is currently in development, which will run on PHP versions greater than 7.0.<br />
<br />
ARK runs on top of these packages and does not provide any set up of them or additional security to them. If you are using your server exposed to the internet, it is essential that you follow good standard security practice for the configuration of your packages. This setup is outside of the scope of this documentation and you are advised to undertake your research carefully.<br />
<br />
In addition the following packages are useful:<br />
<br />
#phpMyAdmin<br />
<br />
Once you have Apache and MySQL running, phpMyAdmin is a very convenient tool for administering databases. More information on how to use phpMyadmin can be found on their [http://docs.phpmyadmin.net/en/latest/user.html website].<br />
<br />
====on LINUX====<br />
<br />
On most Linux distributions these packages will be installed by default.<br />
<br />
=====Apache=====<br />
<br />
Most Linux distributions include an Apache webserver. Consult the documentation on your installation for instructions on how to configure your server. If your service path is configured, the service can be started with:<br />
<br />
<code> sudo service httpd start </code><br />
<br />
If you need to install Apache, instructions can be found on their [http://httpd.apache.org/docs/current/install.html website].<br />
<br />
=====PHP=====<br />
<br />
Most Linux distributions include PHP. Make sure that the version number of PHP installed on your system matches the requirements of ARK.<br />
<br />
If you need to install PHP, instructions can be found on their [http://www.php.net/manual/en/install.php website].<br />
<br />
=====MySQL=====<br />
<br />
MySQL is included in many Linux distributions, but if you do not have it you may need to install <code>mysql-server</code> and <code>mysql</code>, instructions are available on the [http://dev.mysql.com/doc/refman/5.1/en/binary-installation.html MySQL website]. <br />
<br />
You will need to secure the installation, <code>sudo /usr/bin/mysql_secure_installation</code>.<br />
<br />
====on OSX or WINDOWS====<br />
<br />
There are several ways to manually install each of the required packages on OSX or Windows. These methods will work perfectly well with ARK and if you are comfortable installing and configuring the required packages, please go ahead and do so. For OSX the instructions will be similar to the Linux instructions.<br />
<br />
If you would prefer a simple package-based installer, please consider using either WAMP/MAMP or if you are using Windows, MS4W also provides a good way to install a complete WAMP stack along with a Mapserver instance.<br />
<br />
=====MAMP/WAMP=====<br />
<br />
For Mac OSX or Windows, the required packages are available as a bundle known as [http://www.mamp.info/en/index.php MAMP] for Mac or [http://www.wampserver.com/en/ WAMP] Windows. Their websites detail the installation procedures for these services.<br />
<br />
=====MS4W=====<br />
<br />
This package provides a standalone WAMP stack with a Mapserver instance. We have provided some walk through guidance [http://ark.lparchaeology.com/installing-mapserver-and-ms4w/ on the ARK website] for installing this package.<br />
<br />
==Unpack ARK==<br />
<br />
The ARK download consists of a folder of containing php code and several 'config' folders - the config folders contain the SQL code for that configuration. The first step is to copy the arkv1_1 folder into the desired location in your web server directory. Consult the documentation of your distribution if you are unsure where this is. It is likely to be something like <code>/var/www/</code> or <code>/srv/www/</code>. You will then need to either select one of the preconfigured configs and rename it to 'config' or create your own.<br />
<br />
You can now rename the ARK package's folder to your desired name. If you do so, make a note of the folder name because you will need to update the settings to match it. This folder name will be used to navigate your ARK using your browser (in the URL), so call it something short, easy to remember and unique to your installation.<br />
<br />
==Create Database and User==<br />
<br />
You now need to create a new database, import the SQL into it and create a new user on your MySQL server.<br />
<br />
The basic ARK database is included in the ARK package as an SQL file. <br />
<br />
ARK only requires one user on the database, this is the user that the php code will use to access the MySQL database. Actual end users of your ARK will be created later using the tools provided by ARK.<br />
<br />
The easiest way to accomplish these steps is to use phpMyAdmin, but you can also accomplish it using the command line mysql tools:<br />
<br />
===Using phpMyAdmin===<br />
<br />
Use the phpMyAdmin interface to create the new database. This is the database that will contain your ARK data, so make a note of the name so that you can update the settings files to match.<br />
<br />
Now create a new user and grant them privileges on the new database.<br />
<br />
Import the arkv1_1.sql file from the ARK source code into your database. Select the newly created database and then go to the Import tab and Browse to the .zip file that contains the SQL statement. This will copy all of the relevant tables to the database.<br />
<br />
===Using the Command Line===<br />
<br />
You can use MySQL from the command line to add the 'arkuser' user to the MySQL installation for instance: <br />
<code>CREATE USER arkuser'@'%' IDENTIFIED BY 'XXXXXX';<br />
GRANT ALL PRIVILEGES ON *.* TO 'arkuser'@'%' WITH GRANT OPTION;</code><br />
<br />
The values in this must be changed in order to secure your database. See the [http://dev.mysql.com/doc/ mysql manual] for more information. <br />
<br />
You will also need to create a database for your ARK to connect to. <br />
<code>CREATE DATABASE arkv1_1</code><br />
<br />
You are encouraged to change these default names see the [http://dev.mysql.com/doc/ mysql manual] for more information.<br />
<br />
You should now load the ARK SQL into your database by using the <code>Source</code> command on the mysql terminal specifying the location of the arkv1_1.sql file.<br />
<br />
===Use install_ark===<br />
With ARK1.2 there is a bash script which will follow the steps above. This is the quickest way to set up an ARK, but you may wish to change the username or password from the suggested defaults this script will use.<br />
<br />
==Initial Setup==<br />
<br />
===Config===<br />
<br />
You will need to edit some of the configuration directives in the settings files located inside the <code>config</code> folder of your ARK installation.<br />
<br />
====settings====<br />
<br />
Start with the general <code>settings.php</code> file.<br />
<br />
You can update the name of your ARK, this value will be used as the markup nickname (nname) for a piece of markup containing the human readable name of your ARK. This allows you to customise the name of your ARK as it appears in page headers and titles. You will now need to add the markup to the ARK database. <br />
<br />
You should also set your default site code in this file. Make a note of your site code(s) and be sure to add them to the ARK database using the form provided in the data entry page as soon as you log in. You will now need to add the site code to the ARK database.<br />
<br />
====environment settings====<br />
<br />
Next edit your <code>env_settings.php</code> file. This file should be self-explanatory, however there is an explanation of the relevant settings [http://ark.lparchaeology.com/wiki/index.php/Env_settings.php on this wiki page].<br />
<br />
Change the <code>$server</code> directive on line 52 to match your server type, either mac, windows or linux. <br />
<br />
Then in the correct code block of your server type, alter the paths to match the locations of the relevant directories in your ARK environment.<br />
<br />
Make sure that the various path directives are correct. Be aware that you should update these to match your new ark folder name as set up earlier. For example:<br />
<br />
<pre><br />
// The folder name of THIS instance of ARK (relative to the domain)<br />
$ark_dir<br />
// The server path to the ark directory<br />
$ark_server_path<br />
// The path to the PEAR installation<br />
$pear_path<br />
</pre><br />
<br />
Both local and remote paths for OpenLayers are offered, use // to comment out the line that you are not using.<br />
<br />
Alter the database settings at the end of the file to match the database and user that you created earlier.<br />
<br />
<b>You are now ready to connect to your ARK!</b><br />
<br />
===Preflight Checks===<br />
<br />
Point your browser to the preflight checks file included in ARK to test that you have configured ARK correctly. It should be available on your server at www.example.com/'your ark directory'/config/preflight_checks.php. Depending on your setup you may be able to access it at localhost. Check your server documentation for details.<br />
<br />
This tool will list set up options with either PASS or FAIL. If your config FAILs please follow the instructions on the preflight checks page, or refer to other pages on the wiki for more details.<br />
<br />
====File Uploads and phMagick====<br />
<br />
It is highly likely that a standard ARK install will fail on some or all of the tests in this section of the pre-flight tests.<br />
<br />
If you intend to upload files such as PDFs or images to your ARK and would like them to have thumbnails, then you will need to pass all of these tests. If not, then ignore this section of the tests.<br />
<br />
For further information on how to install the required PHP extensions, imageMagick and phMagick, please consult the relevant online documentation.<br />
<br />
===Browse to your ARK===<br />
<br />
Point your browser to the newly created ARK directory, typically this would be <code>www.example.com/ark_dir/</code> or <code>localhost/ark_dir/</code> although the exact URL will vary according to both your hostname and your ARK directory settings.<br />
<br />
After browsing to the ARK directory you should see the login pages for the new ARK system. The default ARK is created with the following admin user account:<br />
<br />
<pre><br />
username: doe_jd<br />
password: janedoe<br />
</pre><br />
<br />
Log in using these details and then finalise your configuration.<br />
<br />
===Final Configuration===<br />
<br />
If you successfully logged in as Jane Doe, pat yourself on the back and now mask sure you finish the last few steps:<br />
<br />
You now need to [http://ark.lparchaeology.com/wiki/index.php/User_Administration create a new user] for yourself on ARK and make yourself and administrator. Once you have created this new administrator, you should logout and log back in again using you new username.<br />
<br />
You <b>should</b> now remove the default doe_jd user to prevent unauthorised access to your system.<br />
<br />
Now [http://ark.lparchaeology.com/wiki/index.php/Markup_Administration add markup] for the ARK name you specified in the settings file.<br />
<br />
Now [http://ark.lparchaeology.com/wiki/index.php/site_code_administration add your default site code] as specified in the settings file.<br />
<br />
==Debug==<br />
<br />
If you experience problems after following the steps above, there are a range of possible reasons that your local setup may not be correct. In order to display debug messages, go to your settings file and turn on debug by adjusting the settings from:<br />
<br />
<pre><br />
error_reporting(0); // Turn off all error reporting - USE THIS FOR PRODUCTION SITES<br />
//error_reporting(E_ALL); // Report all PHP errors (see changelog) - USE THIS FOR SETUP AND TESTING<br />
</pre><br />
<br />
to<br />
<br />
<pre><br />
//error_reporting(0); // Turn off all error reporting - USE THIS FOR PRODUCTION SITES<br />
error_reporting(E_ALL); // Report all PHP errors (see changelog) - USE THIS FOR SETUP AND TESTING<br />
</pre><br />
<br />
==Cleanup==<br />
<br />
Once you have passed the pre-flight checks you should remove (delete or make inaccessible) this file so that it cannot be seen by unwanted users.<br />
<br />
Remove the Jane Doe default user. This is to prevent unauthorised access to your ARK via this account!<br />
<br />
If you plan on using your ARK on an unsecured network (such as the Internet) you should consider how you intend to encrypt traffic between the users and the server. You should consider how you will be securing passwords and login credentials against various kind of attacks. Such security measures are important but are outside the scope of this install document.<br />
<br />
[[category:Administrator]]</div>Mikehttps://ark.lparchaeology.com/wiki/index.php?title=Basic_Installation&diff=124879Basic Installation2018-03-19T12:27:52Z<p>Mike: /* Unpack ARK */</p>
<hr />
<div>If you are comfortable setting up an Apache webserver and a MySQL database, then setting up ARK is very easy. Follow the guide below to get your ARK instance installed and set up.<br />
<br />
If you are unfamiliar with the setup and administration of webservers along with the security risks to your system and data this entails, please consider contacting the ARK team to acquire a fully running ARK.<br />
<br />
There are some further resources for learning about ARK and how to install it and some of the required dependencies [http://ark.lparchaeology.com/resources/ on the ARK website].<br />
<br />
===Download source===<br />
<br />
Download the latest source code from http://sourceforge.net/projects/arkdb/ sourceforge.<br />
<br />
===Check and Install Dependencies===<br />
<br />
====Dependencies====<br />
<br />
ARK requires the following open source software packages to be installed on your server:<br />
<br />
#Apache<br />
#PHP > 5.4 and < 7.0<br />
#MySQL<br />
<br />
ARK is able to connect to any WMC/WFS compliant server of geographic information. Mapserver is no longer an ARK dependency. This means that that administrators can connect to a geographic database of their choice, to suit their needs. Details on [http://mapserver.org/ Mapserver] setup can be found on their website.<br />
<br />
ARK 1.1 requires PHP 5.4 or later. PHP 5.5 will run ARK but will produce "deprecated" errors as some functions used in ARK have been deprecated as of 5.5. If you are using PHP 5.5 and see deprecated messages, these can be hidden by following the relevant PHP documentation. These functions will be updated in a future release of ARK. ARK 1.X will not run on PHP 7.0 or later. ARK 2 is currently in development, which will run on PHP versions greater than 7.0.<br />
<br />
ARK runs on top of these packages and does not provide any set up of them or additional security to them. If you are using your server exposed to the internet, it is essential that you follow good standard security practice for the configuration of your packages. This setup is outside of the scope of this documentation and you are advised to undertake your research carefully.<br />
<br />
In addition the following packages are useful:<br />
<br />
#phpMyAdmin<br />
<br />
Once you have Apache and MySQL running, phpMyAdmin is a very convenient tool for administering databases. More information on how to use phpMyadmin can be found on their [http://docs.phpmyadmin.net/en/latest/user.html website].<br />
<br />
====on LINUX====<br />
<br />
On most Linux distributions these packages will be installed by default.<br />
<br />
=====Apache=====<br />
<br />
Most Linux distributions include an Apache webserver. Consult the documentation on your installation for instructions on how to configure your server. If your service path is configured, the service can be started with:<br />
<br />
<code> sudo service httpd start </code><br />
<br />
If you need to install Apache, instructions can be found on their [http://httpd.apache.org/docs/current/install.html website].<br />
<br />
=====PHP=====<br />
<br />
Most Linux distributions include PHP. Make sure that the version number of PHP installed on your system matches the requirements of ARK.<br />
<br />
If you need to install PHP, instructions can be found on their [http://www.php.net/manual/en/install.php website].<br />
<br />
=====MySQL=====<br />
<br />
MySQL is included in many Linux distributions, but if you do not have it you may need to install <code>mysql-server</code> and <code>mysql</code>, instructions are available on the [http://dev.mysql.com/doc/refman/5.1/en/binary-installation.html MySQL website]. <br />
<br />
You will need to secure the installation, <code>sudo /usr/bin/mysql_secure_installation</code>.<br />
<br />
====on OSX or WINDOWS====<br />
<br />
There are several ways to manually install each of the required packages on OSX or Windows. These methods will work perfectly well with ARK and if you are comfortable installing and configuring the required packages, please go ahead and do so. For OSX the instructions will be similar to the Linux instructions.<br />
<br />
If you would prefer a simple package-based installer, please consider using either WAMP/MAMP or if you are using Windows, MS4W also provides a good way to install a complete WAMP stack along with a Mapserver instance.<br />
<br />
=====MAMP/WAMP=====<br />
<br />
For Mac OSX or Windows, the required packages are available as a bundle known as [http://www.mamp.info/en/index.php MAMP] for Mac or [http://www.wampserver.com/en/ WAMP] Windows. Their websites detail the installation procedures for these services.<br />
<br />
=====MS4W=====<br />
<br />
This package provides a standalone WAMP stack with a Mapserver instance. We have provided some walk through guidance [http://ark.lparchaeology.com/installing-mapserver-and-ms4w/ on the ARK website] for installing this package.<br />
<br />
==Unpack ARK==<br />
<br />
The ARK download consists of a folder of containing php code and several 'config' folders - the config folders contain the SQL code for that configuration. The first step is to copy the arkv1_1 folder into the desired location in your web server directory. Consult the documentation of your distribution if you are unsure where this is. It is likely to be something like <code>/var/www/</code> or <code>/srv/www/</code>. You will then need to either select one of the preconfigured configs and rename it to 'config' or create your own.<br />
<br />
You can now rename the ARK package's folder to your desired name. If you do so, make a note of the folder name because you will need to update the settings to match it. This folder name will be used to navigate your ARK using your browser (in the URL), so call it something short, easy to remember and unique to your installation.<br />
<br />
==Create Database and User==<br />
<br />
You now need to create a new database, import the SQL into it and create a new user on your MySQL server.<br />
<br />
The basic ARK database is included in the ARK package as an SQL file. <br />
<br />
ARK only requires one user on the database, this is the user that the php code will use to access the MySQL database. Actual end users of your ARK will be created later using the tools provided by ARK.<br />
<br />
The easiest way to accomplish these steps is to use phpMyAdmin, but you can also accomplish it using the command line mysql tools:<br />
<br />
===Using phpMyAdmin===<br />
<br />
Use the phpMyAdmin interface to create the new database. This is the database that will contain your ARK data, so make a note of the name so that you can update the settings files to match.<br />
<br />
Now create a new user and grant them privileges on the new database.<br />
<br />
Import the arkv1_1.sql file from the ARK source code into your database. Select the newly created database and then go to the Import tab and Browse to the .zip file that contains the SQL statement. This will copy all of the relevant tables to the database.<br />
<br />
===Using the Command Line===<br />
<br />
You can use MySQL from the command line to add the 'arkuser' user to the MySQL installation for instance: <br />
<code>CREATE USER arkuser'@'%' IDENTIFIED BY 'XXXXXX';<br />
GRANT ALL PRIVILEGES ON *.* TO 'arkuser'@'%' WITH GRANT OPTION;</code><br />
<br />
The values in this must be changed in order to secure your database. See the [http://dev.mysql.com/doc/ mysql manual] for more information. <br />
<br />
You will also need to create a database for your ARK to connect to. <br />
<code>CREATE DATABASE arkv1_1</code><br />
<br />
You are encouraged to change these default names see the [http://dev.mysql.com/doc/ mysql manual] for more information.<br />
<br />
You should now load the ARK SQL into your database by using the <code>Source</code> command on the mysql terminal specifying the location of the arkv1_1.sql file.<br />
<br />
==Initial Setup==<br />
<br />
===Config===<br />
<br />
You will need to edit some of the configuration directives in the settings files located inside the <code>config</code> folder of your ARK installation.<br />
<br />
====settings====<br />
<br />
Start with the general <code>settings.php</code> file.<br />
<br />
You can update the name of your ARK, this value will be used as the markup nickname (nname) for a piece of markup containing the human readable name of your ARK. This allows you to customise the name of your ARK as it appears in page headers and titles. You will now need to add the markup to the ARK database. <br />
<br />
You should also set your default site code in this file. Make a note of your site code(s) and be sure to add them to the ARK database using the form provided in the data entry page as soon as you log in. You will now need to add the site code to the ARK database.<br />
<br />
====environment settings====<br />
<br />
Next edit your <code>env_settings.php</code> file. This file should be self-explanatory, however there is an explanation of the relevant settings [http://ark.lparchaeology.com/wiki/index.php/Env_settings.php on this wiki page].<br />
<br />
Change the <code>$server</code> directive on line 52 to match your server type, either mac, windows or linux. <br />
<br />
Then in the correct code block of your server type, alter the paths to match the locations of the relevant directories in your ARK environment.<br />
<br />
Make sure that the various path directives are correct. Be aware that you should update these to match your new ark folder name as set up earlier. For example:<br />
<br />
<pre><br />
// The folder name of THIS instance of ARK (relative to the domain)<br />
$ark_dir<br />
// The server path to the ark directory<br />
$ark_server_path<br />
// The path to the PEAR installation<br />
$pear_path<br />
</pre><br />
<br />
Both local and remote paths for OpenLayers are offered, use // to comment out the line that you are not using.<br />
<br />
Alter the database settings at the end of the file to match the database and user that you created earlier.<br />
<br />
<b>You are now ready to connect to your ARK!</b><br />
<br />
===Preflight Checks===<br />
<br />
Point your browser to the preflight checks file included in ARK to test that you have configured ARK correctly. It should be available on your server at www.example.com/'your ark directory'/config/preflight_checks.php. Depending on your setup you may be able to access it at localhost. Check your server documentation for details.<br />
<br />
This tool will list set up options with either PASS or FAIL. If your config FAILs please follow the instructions on the preflight checks page, or refer to other pages on the wiki for more details.<br />
<br />
====File Uploads and phMagick====<br />
<br />
It is highly likely that a standard ARK install will fail on some or all of the tests in this section of the pre-flight tests.<br />
<br />
If you intend to upload files such as PDFs or images to your ARK and would like them to have thumbnails, then you will need to pass all of these tests. If not, then ignore this section of the tests.<br />
<br />
For further information on how to install the required PHP extensions, imageMagick and phMagick, please consult the relevant online documentation.<br />
<br />
===Browse to your ARK===<br />
<br />
Point your browser to the newly created ARK directory, typically this would be <code>www.example.com/ark_dir/</code> or <code>localhost/ark_dir/</code> although the exact URL will vary according to both your hostname and your ARK directory settings.<br />
<br />
After browsing to the ARK directory you should see the login pages for the new ARK system. The default ARK is created with the following admin user account:<br />
<br />
<pre><br />
username: doe_jd<br />
password: janedoe<br />
</pre><br />
<br />
Log in using these details and then finalise your configuration.<br />
<br />
===Final Configuration===<br />
<br />
If you successfully logged in as Jane Doe, pat yourself on the back and now mask sure you finish the last few steps:<br />
<br />
You now need to [http://ark.lparchaeology.com/wiki/index.php/User_Administration create a new user] for yourself on ARK and make yourself and administrator. Once you have created this new administrator, you should logout and log back in again using you new username.<br />
<br />
You <b>should</b> now remove the default doe_jd user to prevent unauthorised access to your system.<br />
<br />
Now [http://ark.lparchaeology.com/wiki/index.php/Markup_Administration add markup] for the ARK name you specified in the settings file.<br />
<br />
Now [http://ark.lparchaeology.com/wiki/index.php/site_code_administration add your default site code] as specified in the settings file.<br />
<br />
==Debug==<br />
<br />
If you experience problems after following the steps above, there are a range of possible reasons that your local setup may not be correct. In order to display debug messages, go to your settings file and turn on debug by adjusting the settings from:<br />
<br />
<pre><br />
error_reporting(0); // Turn off all error reporting - USE THIS FOR PRODUCTION SITES<br />
//error_reporting(E_ALL); // Report all PHP errors (see changelog) - USE THIS FOR SETUP AND TESTING<br />
</pre><br />
<br />
to<br />
<br />
<pre><br />
//error_reporting(0); // Turn off all error reporting - USE THIS FOR PRODUCTION SITES<br />
error_reporting(E_ALL); // Report all PHP errors (see changelog) - USE THIS FOR SETUP AND TESTING<br />
</pre><br />
<br />
==Cleanup==<br />
<br />
Once you have passed the pre-flight checks you should remove (delete or make inaccessible) this file so that it cannot be seen by unwanted users.<br />
<br />
Remove the Jane Doe default user. This is to prevent unauthorised access to your ARK via this account!<br />
<br />
If you plan on using your ARK on an unsecured network (such as the Internet) you should consider how you intend to encrypt traffic between the users and the server. You should consider how you will be securing passwords and login credentials against various kind of attacks. Such security measures are important but are outside the scope of this install document.<br />
<br />
[[category:Administrator]]</div>Mikehttps://ark.lparchaeology.com/wiki/index.php?title=Basic_Installation&diff=124878Basic Installation2018-03-19T12:26:47Z<p>Mike: /* Unpack ARK */</p>
<hr />
<div>If you are comfortable setting up an Apache webserver and a MySQL database, then setting up ARK is very easy. Follow the guide below to get your ARK instance installed and set up.<br />
<br />
If you are unfamiliar with the setup and administration of webservers along with the security risks to your system and data this entails, please consider contacting the ARK team to acquire a fully running ARK.<br />
<br />
There are some further resources for learning about ARK and how to install it and some of the required dependencies [http://ark.lparchaeology.com/resources/ on the ARK website].<br />
<br />
===Download source===<br />
<br />
Download the latest source code from http://sourceforge.net/projects/arkdb/ sourceforge.<br />
<br />
===Check and Install Dependencies===<br />
<br />
====Dependencies====<br />
<br />
ARK requires the following open source software packages to be installed on your server:<br />
<br />
#Apache<br />
#PHP > 5.4 and < 7.0<br />
#MySQL<br />
<br />
ARK is able to connect to any WMC/WFS compliant server of geographic information. Mapserver is no longer an ARK dependency. This means that that administrators can connect to a geographic database of their choice, to suit their needs. Details on [http://mapserver.org/ Mapserver] setup can be found on their website.<br />
<br />
ARK 1.1 requires PHP 5.4 or later. PHP 5.5 will run ARK but will produce "deprecated" errors as some functions used in ARK have been deprecated as of 5.5. If you are using PHP 5.5 and see deprecated messages, these can be hidden by following the relevant PHP documentation. These functions will be updated in a future release of ARK. ARK 1.X will not run on PHP 7.0 or later. ARK 2 is currently in development, which will run on PHP versions greater than 7.0.<br />
<br />
ARK runs on top of these packages and does not provide any set up of them or additional security to them. If you are using your server exposed to the internet, it is essential that you follow good standard security practice for the configuration of your packages. This setup is outside of the scope of this documentation and you are advised to undertake your research carefully.<br />
<br />
In addition the following packages are useful:<br />
<br />
#phpMyAdmin<br />
<br />
Once you have Apache and MySQL running, phpMyAdmin is a very convenient tool for administering databases. More information on how to use phpMyadmin can be found on their [http://docs.phpmyadmin.net/en/latest/user.html website].<br />
<br />
====on LINUX====<br />
<br />
On most Linux distributions these packages will be installed by default.<br />
<br />
=====Apache=====<br />
<br />
Most Linux distributions include an Apache webserver. Consult the documentation on your installation for instructions on how to configure your server. If your service path is configured, the service can be started with:<br />
<br />
<code> sudo service httpd start </code><br />
<br />
If you need to install Apache, instructions can be found on their [http://httpd.apache.org/docs/current/install.html website].<br />
<br />
=====PHP=====<br />
<br />
Most Linux distributions include PHP. Make sure that the version number of PHP installed on your system matches the requirements of ARK.<br />
<br />
If you need to install PHP, instructions can be found on their [http://www.php.net/manual/en/install.php website].<br />
<br />
=====MySQL=====<br />
<br />
MySQL is included in many Linux distributions, but if you do not have it you may need to install <code>mysql-server</code> and <code>mysql</code>, instructions are available on the [http://dev.mysql.com/doc/refman/5.1/en/binary-installation.html MySQL website]. <br />
<br />
You will need to secure the installation, <code>sudo /usr/bin/mysql_secure_installation</code>.<br />
<br />
====on OSX or WINDOWS====<br />
<br />
There are several ways to manually install each of the required packages on OSX or Windows. These methods will work perfectly well with ARK and if you are comfortable installing and configuring the required packages, please go ahead and do so. For OSX the instructions will be similar to the Linux instructions.<br />
<br />
If you would prefer a simple package-based installer, please consider using either WAMP/MAMP or if you are using Windows, MS4W also provides a good way to install a complete WAMP stack along with a Mapserver instance.<br />
<br />
=====MAMP/WAMP=====<br />
<br />
For Mac OSX or Windows, the required packages are available as a bundle known as [http://www.mamp.info/en/index.php MAMP] for Mac or [http://www.wampserver.com/en/ WAMP] Windows. Their websites detail the installation procedures for these services.<br />
<br />
=====MS4W=====<br />
<br />
This package provides a standalone WAMP stack with a Mapserver instance. We have provided some walk through guidance [http://ark.lparchaeology.com/installing-mapserver-and-ms4w/ on the ARK website] for installing this package.<br />
<br />
==Unpack ARK==<br />
<br />
The ARK download consists of a folder of containing php code and several 'config' folders - the config folders contain the SQL code for that configuration. The first step is to copy the arkv1_1 folder into the desired location in your web server directory. Consult the documentation of your distribution if you are unsure where this is. It is likely to be something like <code>/var/www/</code> or <code>/srv/www/</code>.<br />
<br />
You can now rename the ARK package's folder to your desired name. If you do so, make a note of the folder name because you will need to update the settings to match it. This folder name will be used to navigate your ARK using your browser (in the URL), so call it something short, easy to remember and unique to your installation.<br />
<br />
==Create Database and User==<br />
<br />
You now need to create a new database, import the SQL into it and create a new user on your MySQL server.<br />
<br />
The basic ARK database is included in the ARK package as an SQL file. <br />
<br />
ARK only requires one user on the database, this is the user that the php code will use to access the MySQL database. Actual end users of your ARK will be created later using the tools provided by ARK.<br />
<br />
The easiest way to accomplish these steps is to use phpMyAdmin, but you can also accomplish it using the command line mysql tools:<br />
<br />
===Using phpMyAdmin===<br />
<br />
Use the phpMyAdmin interface to create the new database. This is the database that will contain your ARK data, so make a note of the name so that you can update the settings files to match.<br />
<br />
Now create a new user and grant them privileges on the new database.<br />
<br />
Import the arkv1_1.sql file from the ARK source code into your database. Select the newly created database and then go to the Import tab and Browse to the .zip file that contains the SQL statement. This will copy all of the relevant tables to the database.<br />
<br />
===Using the Command Line===<br />
<br />
You can use MySQL from the command line to add the 'arkuser' user to the MySQL installation for instance: <br />
<code>CREATE USER arkuser'@'%' IDENTIFIED BY 'XXXXXX';<br />
GRANT ALL PRIVILEGES ON *.* TO 'arkuser'@'%' WITH GRANT OPTION;</code><br />
<br />
The values in this must be changed in order to secure your database. See the [http://dev.mysql.com/doc/ mysql manual] for more information. <br />
<br />
You will also need to create a database for your ARK to connect to. <br />
<code>CREATE DATABASE arkv1_1</code><br />
<br />
You are encouraged to change these default names see the [http://dev.mysql.com/doc/ mysql manual] for more information.<br />
<br />
You should now load the ARK SQL into your database by using the <code>Source</code> command on the mysql terminal specifying the location of the arkv1_1.sql file.<br />
<br />
==Initial Setup==<br />
<br />
===Config===<br />
<br />
You will need to edit some of the configuration directives in the settings files located inside the <code>config</code> folder of your ARK installation.<br />
<br />
====settings====<br />
<br />
Start with the general <code>settings.php</code> file.<br />
<br />
You can update the name of your ARK, this value will be used as the markup nickname (nname) for a piece of markup containing the human readable name of your ARK. This allows you to customise the name of your ARK as it appears in page headers and titles. You will now need to add the markup to the ARK database. <br />
<br />
You should also set your default site code in this file. Make a note of your site code(s) and be sure to add them to the ARK database using the form provided in the data entry page as soon as you log in. You will now need to add the site code to the ARK database.<br />
<br />
====environment settings====<br />
<br />
Next edit your <code>env_settings.php</code> file. This file should be self-explanatory, however there is an explanation of the relevant settings [http://ark.lparchaeology.com/wiki/index.php/Env_settings.php on this wiki page].<br />
<br />
Change the <code>$server</code> directive on line 52 to match your server type, either mac, windows or linux. <br />
<br />
Then in the correct code block of your server type, alter the paths to match the locations of the relevant directories in your ARK environment.<br />
<br />
Make sure that the various path directives are correct. Be aware that you should update these to match your new ark folder name as set up earlier. For example:<br />
<br />
<pre><br />
// The folder name of THIS instance of ARK (relative to the domain)<br />
$ark_dir<br />
// The server path to the ark directory<br />
$ark_server_path<br />
// The path to the PEAR installation<br />
$pear_path<br />
</pre><br />
<br />
Both local and remote paths for OpenLayers are offered, use // to comment out the line that you are not using.<br />
<br />
Alter the database settings at the end of the file to match the database and user that you created earlier.<br />
<br />
<b>You are now ready to connect to your ARK!</b><br />
<br />
===Preflight Checks===<br />
<br />
Point your browser to the preflight checks file included in ARK to test that you have configured ARK correctly. It should be available on your server at www.example.com/'your ark directory'/config/preflight_checks.php. Depending on your setup you may be able to access it at localhost. Check your server documentation for details.<br />
<br />
This tool will list set up options with either PASS or FAIL. If your config FAILs please follow the instructions on the preflight checks page, or refer to other pages on the wiki for more details.<br />
<br />
====File Uploads and phMagick====<br />
<br />
It is highly likely that a standard ARK install will fail on some or all of the tests in this section of the pre-flight tests.<br />
<br />
If you intend to upload files such as PDFs or images to your ARK and would like them to have thumbnails, then you will need to pass all of these tests. If not, then ignore this section of the tests.<br />
<br />
For further information on how to install the required PHP extensions, imageMagick and phMagick, please consult the relevant online documentation.<br />
<br />
===Browse to your ARK===<br />
<br />
Point your browser to the newly created ARK directory, typically this would be <code>www.example.com/ark_dir/</code> or <code>localhost/ark_dir/</code> although the exact URL will vary according to both your hostname and your ARK directory settings.<br />
<br />
After browsing to the ARK directory you should see the login pages for the new ARK system. The default ARK is created with the following admin user account:<br />
<br />
<pre><br />
username: doe_jd<br />
password: janedoe<br />
</pre><br />
<br />
Log in using these details and then finalise your configuration.<br />
<br />
===Final Configuration===<br />
<br />
If you successfully logged in as Jane Doe, pat yourself on the back and now mask sure you finish the last few steps:<br />
<br />
You now need to [http://ark.lparchaeology.com/wiki/index.php/User_Administration create a new user] for yourself on ARK and make yourself and administrator. Once you have created this new administrator, you should logout and log back in again using you new username.<br />
<br />
You <b>should</b> now remove the default doe_jd user to prevent unauthorised access to your system.<br />
<br />
Now [http://ark.lparchaeology.com/wiki/index.php/Markup_Administration add markup] for the ARK name you specified in the settings file.<br />
<br />
Now [http://ark.lparchaeology.com/wiki/index.php/site_code_administration add your default site code] as specified in the settings file.<br />
<br />
==Debug==<br />
<br />
If you experience problems after following the steps above, there are a range of possible reasons that your local setup may not be correct. In order to display debug messages, go to your settings file and turn on debug by adjusting the settings from:<br />
<br />
<pre><br />
error_reporting(0); // Turn off all error reporting - USE THIS FOR PRODUCTION SITES<br />
//error_reporting(E_ALL); // Report all PHP errors (see changelog) - USE THIS FOR SETUP AND TESTING<br />
</pre><br />
<br />
to<br />
<br />
<pre><br />
//error_reporting(0); // Turn off all error reporting - USE THIS FOR PRODUCTION SITES<br />
error_reporting(E_ALL); // Report all PHP errors (see changelog) - USE THIS FOR SETUP AND TESTING<br />
</pre><br />
<br />
==Cleanup==<br />
<br />
Once you have passed the pre-flight checks you should remove (delete or make inaccessible) this file so that it cannot be seen by unwanted users.<br />
<br />
Remove the Jane Doe default user. This is to prevent unauthorised access to your ARK via this account!<br />
<br />
If you plan on using your ARK on an unsecured network (such as the Internet) you should consider how you intend to encrypt traffic between the users and the server. You should consider how you will be securing passwords and login credentials against various kind of attacks. Such security measures are important but are outside the scope of this install document.<br />
<br />
[[category:Administrator]]</div>Mikehttps://ark.lparchaeology.com/wiki/index.php?title=Basic_Installation&diff=124877Basic Installation2018-03-19T12:24:51Z<p>Mike: /* Dependencies */</p>
<hr />
<div>If you are comfortable setting up an Apache webserver and a MySQL database, then setting up ARK is very easy. Follow the guide below to get your ARK instance installed and set up.<br />
<br />
If you are unfamiliar with the setup and administration of webservers along with the security risks to your system and data this entails, please consider contacting the ARK team to acquire a fully running ARK.<br />
<br />
There are some further resources for learning about ARK and how to install it and some of the required dependencies [http://ark.lparchaeology.com/resources/ on the ARK website].<br />
<br />
===Download source===<br />
<br />
Download the latest source code from http://sourceforge.net/projects/arkdb/ sourceforge.<br />
<br />
===Check and Install Dependencies===<br />
<br />
====Dependencies====<br />
<br />
ARK requires the following open source software packages to be installed on your server:<br />
<br />
#Apache<br />
#PHP > 5.4 and < 7.0<br />
#MySQL<br />
<br />
ARK is able to connect to any WMC/WFS compliant server of geographic information. Mapserver is no longer an ARK dependency. This means that that administrators can connect to a geographic database of their choice, to suit their needs. Details on [http://mapserver.org/ Mapserver] setup can be found on their website.<br />
<br />
ARK 1.1 requires PHP 5.4 or later. PHP 5.5 will run ARK but will produce "deprecated" errors as some functions used in ARK have been deprecated as of 5.5. If you are using PHP 5.5 and see deprecated messages, these can be hidden by following the relevant PHP documentation. These functions will be updated in a future release of ARK. ARK 1.X will not run on PHP 7.0 or later. ARK 2 is currently in development, which will run on PHP versions greater than 7.0.<br />
<br />
ARK runs on top of these packages and does not provide any set up of them or additional security to them. If you are using your server exposed to the internet, it is essential that you follow good standard security practice for the configuration of your packages. This setup is outside of the scope of this documentation and you are advised to undertake your research carefully.<br />
<br />
In addition the following packages are useful:<br />
<br />
#phpMyAdmin<br />
<br />
Once you have Apache and MySQL running, phpMyAdmin is a very convenient tool for administering databases. More information on how to use phpMyadmin can be found on their [http://docs.phpmyadmin.net/en/latest/user.html website].<br />
<br />
====on LINUX====<br />
<br />
On most Linux distributions these packages will be installed by default.<br />
<br />
=====Apache=====<br />
<br />
Most Linux distributions include an Apache webserver. Consult the documentation on your installation for instructions on how to configure your server. If your service path is configured, the service can be started with:<br />
<br />
<code> sudo service httpd start </code><br />
<br />
If you need to install Apache, instructions can be found on their [http://httpd.apache.org/docs/current/install.html website].<br />
<br />
=====PHP=====<br />
<br />
Most Linux distributions include PHP. Make sure that the version number of PHP installed on your system matches the requirements of ARK.<br />
<br />
If you need to install PHP, instructions can be found on their [http://www.php.net/manual/en/install.php website].<br />
<br />
=====MySQL=====<br />
<br />
MySQL is included in many Linux distributions, but if you do not have it you may need to install <code>mysql-server</code> and <code>mysql</code>, instructions are available on the [http://dev.mysql.com/doc/refman/5.1/en/binary-installation.html MySQL website]. <br />
<br />
You will need to secure the installation, <code>sudo /usr/bin/mysql_secure_installation</code>.<br />
<br />
====on OSX or WINDOWS====<br />
<br />
There are several ways to manually install each of the required packages on OSX or Windows. These methods will work perfectly well with ARK and if you are comfortable installing and configuring the required packages, please go ahead and do so. For OSX the instructions will be similar to the Linux instructions.<br />
<br />
If you would prefer a simple package-based installer, please consider using either WAMP/MAMP or if you are using Windows, MS4W also provides a good way to install a complete WAMP stack along with a Mapserver instance.<br />
<br />
=====MAMP/WAMP=====<br />
<br />
For Mac OSX or Windows, the required packages are available as a bundle known as [http://www.mamp.info/en/index.php MAMP] for Mac or [http://www.wampserver.com/en/ WAMP] Windows. Their websites detail the installation procedures for these services.<br />
<br />
=====MS4W=====<br />
<br />
This package provides a standalone WAMP stack with a Mapserver instance. We have provided some walk through guidance [http://ark.lparchaeology.com/installing-mapserver-and-ms4w/ on the ARK website] for installing this package.<br />
<br />
==Unpack ARK==<br />
<br />
The ARK download consists of an SQL file and a folder of the php code and configuration files. All you need to do is to copy the arkv1_1 folder into the desired location in your web server directory. Consult the documentation of your distribution if you are unsure where this is. It is likely to be something like <code>/var/www/</code> or <code>/srv/www/</code>.<br />
<br />
You can now rename the ARK package's folder to your desired name. If you do so, make a note of the folder name because you will need to update the environment settings to match it. This folder name will be used to navigate your ARK using your browser (in the URL), so call it something short, easy to remember and unique to your installation.<br />
<br />
==Create Database and User==<br />
<br />
You now need to create a new database, import the SQL into it and create a new user on your MySQL server.<br />
<br />
The basic ARK database is included in the ARK package as an SQL file. <br />
<br />
ARK only requires one user on the database, this is the user that the php code will use to access the MySQL database. Actual end users of your ARK will be created later using the tools provided by ARK.<br />
<br />
The easiest way to accomplish these steps is to use phpMyAdmin, but you can also accomplish it using the command line mysql tools:<br />
<br />
===Using phpMyAdmin===<br />
<br />
Use the phpMyAdmin interface to create the new database. This is the database that will contain your ARK data, so make a note of the name so that you can update the settings files to match.<br />
<br />
Now create a new user and grant them privileges on the new database.<br />
<br />
Import the arkv1_1.sql file from the ARK source code into your database. Select the newly created database and then go to the Import tab and Browse to the .zip file that contains the SQL statement. This will copy all of the relevant tables to the database.<br />
<br />
===Using the Command Line===<br />
<br />
You can use MySQL from the command line to add the 'arkuser' user to the MySQL installation for instance: <br />
<code>CREATE USER arkuser'@'%' IDENTIFIED BY 'XXXXXX';<br />
GRANT ALL PRIVILEGES ON *.* TO 'arkuser'@'%' WITH GRANT OPTION;</code><br />
<br />
The values in this must be changed in order to secure your database. See the [http://dev.mysql.com/doc/ mysql manual] for more information. <br />
<br />
You will also need to create a database for your ARK to connect to. <br />
<code>CREATE DATABASE arkv1_1</code><br />
<br />
You are encouraged to change these default names see the [http://dev.mysql.com/doc/ mysql manual] for more information.<br />
<br />
You should now load the ARK SQL into your database by using the <code>Source</code> command on the mysql terminal specifying the location of the arkv1_1.sql file.<br />
<br />
==Initial Setup==<br />
<br />
===Config===<br />
<br />
You will need to edit some of the configuration directives in the settings files located inside the <code>config</code> folder of your ARK installation.<br />
<br />
====settings====<br />
<br />
Start with the general <code>settings.php</code> file.<br />
<br />
You can update the name of your ARK, this value will be used as the markup nickname (nname) for a piece of markup containing the human readable name of your ARK. This allows you to customise the name of your ARK as it appears in page headers and titles. You will now need to add the markup to the ARK database. <br />
<br />
You should also set your default site code in this file. Make a note of your site code(s) and be sure to add them to the ARK database using the form provided in the data entry page as soon as you log in. You will now need to add the site code to the ARK database.<br />
<br />
====environment settings====<br />
<br />
Next edit your <code>env_settings.php</code> file. This file should be self-explanatory, however there is an explanation of the relevant settings [http://ark.lparchaeology.com/wiki/index.php/Env_settings.php on this wiki page].<br />
<br />
Change the <code>$server</code> directive on line 52 to match your server type, either mac, windows or linux. <br />
<br />
Then in the correct code block of your server type, alter the paths to match the locations of the relevant directories in your ARK environment.<br />
<br />
Make sure that the various path directives are correct. Be aware that you should update these to match your new ark folder name as set up earlier. For example:<br />
<br />
<pre><br />
// The folder name of THIS instance of ARK (relative to the domain)<br />
$ark_dir<br />
// The server path to the ark directory<br />
$ark_server_path<br />
// The path to the PEAR installation<br />
$pear_path<br />
</pre><br />
<br />
Both local and remote paths for OpenLayers are offered, use // to comment out the line that you are not using.<br />
<br />
Alter the database settings at the end of the file to match the database and user that you created earlier.<br />
<br />
<b>You are now ready to connect to your ARK!</b><br />
<br />
===Preflight Checks===<br />
<br />
Point your browser to the preflight checks file included in ARK to test that you have configured ARK correctly. It should be available on your server at www.example.com/'your ark directory'/config/preflight_checks.php. Depending on your setup you may be able to access it at localhost. Check your server documentation for details.<br />
<br />
This tool will list set up options with either PASS or FAIL. If your config FAILs please follow the instructions on the preflight checks page, or refer to other pages on the wiki for more details.<br />
<br />
====File Uploads and phMagick====<br />
<br />
It is highly likely that a standard ARK install will fail on some or all of the tests in this section of the pre-flight tests.<br />
<br />
If you intend to upload files such as PDFs or images to your ARK and would like them to have thumbnails, then you will need to pass all of these tests. If not, then ignore this section of the tests.<br />
<br />
For further information on how to install the required PHP extensions, imageMagick and phMagick, please consult the relevant online documentation.<br />
<br />
===Browse to your ARK===<br />
<br />
Point your browser to the newly created ARK directory, typically this would be <code>www.example.com/ark_dir/</code> or <code>localhost/ark_dir/</code> although the exact URL will vary according to both your hostname and your ARK directory settings.<br />
<br />
After browsing to the ARK directory you should see the login pages for the new ARK system. The default ARK is created with the following admin user account:<br />
<br />
<pre><br />
username: doe_jd<br />
password: janedoe<br />
</pre><br />
<br />
Log in using these details and then finalise your configuration.<br />
<br />
===Final Configuration===<br />
<br />
If you successfully logged in as Jane Doe, pat yourself on the back and now mask sure you finish the last few steps:<br />
<br />
You now need to [http://ark.lparchaeology.com/wiki/index.php/User_Administration create a new user] for yourself on ARK and make yourself and administrator. Once you have created this new administrator, you should logout and log back in again using you new username.<br />
<br />
You <b>should</b> now remove the default doe_jd user to prevent unauthorised access to your system.<br />
<br />
Now [http://ark.lparchaeology.com/wiki/index.php/Markup_Administration add markup] for the ARK name you specified in the settings file.<br />
<br />
Now [http://ark.lparchaeology.com/wiki/index.php/site_code_administration add your default site code] as specified in the settings file.<br />
<br />
==Debug==<br />
<br />
If you experience problems after following the steps above, there are a range of possible reasons that your local setup may not be correct. In order to display debug messages, go to your settings file and turn on debug by adjusting the settings from:<br />
<br />
<pre><br />
error_reporting(0); // Turn off all error reporting - USE THIS FOR PRODUCTION SITES<br />
//error_reporting(E_ALL); // Report all PHP errors (see changelog) - USE THIS FOR SETUP AND TESTING<br />
</pre><br />
<br />
to<br />
<br />
<pre><br />
//error_reporting(0); // Turn off all error reporting - USE THIS FOR PRODUCTION SITES<br />
error_reporting(E_ALL); // Report all PHP errors (see changelog) - USE THIS FOR SETUP AND TESTING<br />
</pre><br />
<br />
==Cleanup==<br />
<br />
Once you have passed the pre-flight checks you should remove (delete or make inaccessible) this file so that it cannot be seen by unwanted users.<br />
<br />
Remove the Jane Doe default user. This is to prevent unauthorised access to your ARK via this account!<br />
<br />
If you plan on using your ARK on an unsecured network (such as the Internet) you should consider how you intend to encrypt traffic between the users and the server. You should consider how you will be securing passwords and login credentials against various kind of attacks. Such security measures are important but are outside the scope of this install document.<br />
<br />
[[category:Administrator]]</div>Mikehttps://ark.lparchaeology.com/wiki/index.php?title=Available_config&diff=124812Available config2017-11-27T16:13:09Z<p>Mike: /* ARK Empty */</p>
<hr />
<div><br />
With the ARK1.2 release there are several available default setups or "configs" for ARK. When installing ARK you must choose the right one for your project.<br />
<br />
===Single Context===<br />
This is L - P : Archaeology's favourite config, it is a full implementation of the MoL recording system including modules for Contexts, Sections, Plans, Registered Finds, Samples, Site Photographs, Timber, 'Trenches, Areas, Etc' (TAE), Subgroups, Groups and Landuses.<br />
<br />
===Basic Context===<br />
This is a simple version of the single context recording system which has modules for Contexts, Plans, Sections, Registered Finds and Site Photos.<br />
<br />
===ARK Landscape===<br />
This config is good for landscape studies it has modules for Locations, Excavations, Finds, Catalogued Finds, Illustrations and Photos.<br />
<br />
===ARK Stratigraphic Unit===<br />
This config is used by some mediterranean projects, it has modules for Contexts, Site Photos, Plans, Geophotos, Architectural Elements, Special Finds and Coins<br />
<br />
===ARK Empty===<br />
ARK is endlessly customisable, the empty config has only the required addressbook module, allowing you to create your own config with modules from the other configs, or by creating them yourself. check out the [http://ark.lparchaeology.com/wiki/index.php/Mod_settings.php Mod Settings] page to see how.</div>Mikehttps://ark.lparchaeology.com/wiki/index.php?title=Available_config&diff=124811Available config2017-11-27T16:10:16Z<p>Mike: Created page with " With the ARK1.2 release there are several available default setups or "configs" for ARK. When installing ARK you must choose the right one for your project. ===Single Contex..."</p>
<hr />
<div><br />
With the ARK1.2 release there are several available default setups or "configs" for ARK. When installing ARK you must choose the right one for your project.<br />
<br />
===Single Context===<br />
This is L - P : Archaeology's favourite config, it is a full implementation of the MoL recording system including modules for Contexts, Sections, Plans, Registered Finds, Samples, Site Photographs, Timber, 'Trenches, Areas, Etc' (TAE), Subgroups, Groups and Landuses.<br />
<br />
===Basic Context===<br />
This is a simple version of the single context recording system which has modules for Contexts, Plans, Sections, Registered Finds and Site Photos.<br />
<br />
===ARK Landscape===<br />
This config is good for landscape studies it has modules for Locations, Excavations, Finds, Catalogued Finds, Illustrations and Photos.<br />
<br />
===ARK Stratigraphic Unit===<br />
This config is used by some mediterranean projects, it has modules for Contexts, Site Photos, Plans, Geophotos, Architectural Elements, Special Finds and Coins<br />
<br />
===ARK Empty===<br />
ARK is endlessly customisable, the empty config has only the required addressbook module, allowing you to create your own config with modules from the other configs, or by creating them yourself. check out the [[http://ark.lparchaeology.com/wiki/index.php/Mod_settings.php]] to see how.</div>Mikehttps://ark.lparchaeology.com/wiki/index.php?title=ARK2/Technical&diff=124651ARK2/Technical2016-12-19T15:52:12Z<p>Mike: /* macOS */</p>
<hr />
<div>= Technical Details =<br />
<br />
== Platforms ==<br />
<br />
* HTML5 will be used<br />
* Browser support restricted to those supported by Bootstrap 3<br />
* PHP: A minimum of v5.6 will be supported (5.6 is in Security Support, 5.7 in active support, see http://php.net/supported-versions.php), v7 will be supported.<br />
* MySQL/MariaDB v5.5 or later (lowest supported MySQL) (5.6 for full-text index, 5.7.5 for spatial index)<br />
* PostgreSQL v9.2 or later<br />
* SQLite 3.7.11 or later (required for multiple inserts)<br />
* mod_rewrite will be required<br />
* PHP intl extension and ICU will be required<br />
<br />
== Standards ==<br />
<br />
The [http://www.php-fig.org/psr/ PHP-FIG standards] will be used:<br />
* PSR-1 and PSR-2 Coding Standards<br />
* PSR-3 Logging Interface for interchangeable logging objects<br />
* PSR-4 Auto-Loading Standard<br />
* PSR-5 PHPDoc Standard (proposed)<br />
* Symfony HTTP Foundation for interchangeable Request/Response objects<br />
* PSR-7 HTTP Message Interface for interchangeable Request/Response objects if required for some components<br />
* All files will be UTF8 using UNIX LF<br />
<br />
Plugins are available for Eclipse, Atom, and other IDEs to automatically check and/or apply these standards.<br />
<br />
== Framework and Components ==<br />
<br />
The chosen framework is the [http://silex.sensiolabs.org/ Silex] micro-framework built using [http://symfony.com/ Symphony] components<br />
<br />
Components will be carefully chosen to be well supported, stable, and interchangeable wherever possible. A number of criteria will be applied in selection:<br />
* Must be standards compliant<br />
* Must be well supported with a solid development history<br />
* Must be well documented<br />
* Must be widely used and supported<br />
* Must have a strong community, small one-person efforts will only be considered if they are the de-facto standard<br />
* Any database access must use Doctrine DBAL or ORM<br />
<br />
Significant and reliable sources of components include:<br />
* [https://packagist.org/ Packagist], the Composer index <br />
* [http://symfony.com/components Symfony]<br />
* [http://thephpleague.com/#packages PHP League]<br />
* [https://zendframework.github.io/ Zend]<br />
* [http://docs.sylius.org/en/latest/components/index.html Sylius], components from an e-commerce platform based on Symfony<br />
* [http://auraphp.com/ Aura]<br />
* [https://github.com/silexphp/Silex/wiki/Third-Party-ServiceProviders-for-Silex-2.x Silex providers]<br />
<br />
* [http://www.userfrosting.com/ User frosting], a UserCake fork with RBAC user management, using Slim2, SBAdmin2, use for ideas<br />
<br />
== PHP Autoloading ==<br />
<br />
PSR-4 is used for packaging, namespace and auto-loading of OO code.<br />
* [https://getcomposer.org/ Composer] will be required for dependency management and PSR-4 auto-loading<br />
* All external libraries will be installed by Composer under vendor/<br />
* All OO classes will be namespaced under ARK\<br />
* All OO code will be under src/<br />
<br />
A good series of articles explaining PSR-4 and modern development and packaging in general can be found at the following:<br />
* http://culttt.com/2013/01/07/what-is-php-composer/<br />
* http://culttt.com/2014/03/12/build-php-package/<br />
* http://culttt.com/2014/05/07/create-psr-4-php-package/<br />
<br />
== Git / Github ==<br />
<br />
Development is hosted on [https://github.com/lparchaeology Github], so you will need a free Github account to contribute code.<br />
<br />
If you are new to Git, you may find the [https://desktop.github.com/ Github desktop application] easier to use. Alternatives are [https://www.sourcetreeapp.com/ SourceTree] and [https://www.gitkraken.com/ GitKraken].<br />
<br />
We will use a simplifed version of the [https://wiki.qt.io/Branch_Guidelines Qt git workflow].<br />
* '''dev''' branch is the semi-stable development branch for the next release<br />
* New feature or bugfix branches are branched off '''dev''' with the name of the author/group/client and the feature or bug number, e.g. '''jlayt/linked-data'''<br />
* Once a new feature is finished and passes testing, it is rebased onto the current '''dev''' and merged via a Github merge request with review<br />
* Alpha and Beta testing releases will be tagged on '''dev''' and not branched<br />
* Once '''dev''' is deemed feature complete and stable enough for a Release Candidate then a release branch will be branched off '''dev''' with the minor release number, e.g. '''release/2.0''', '''release/2.1''', etc<br />
* Further testing will take place on the release branch, with fixes applied to to the release branch as required<br />
* Once ready for release, the release will be tagged with the point release number (e.g. 2.0.0) and all bugfixes merged back into '''dev'''<br />
* Any bugfixes for the stable release will be made in the earliest release branch they apply to and then merged forward through each release then eventually into '''dev''' (merges may be immediate or periodic depending on the volume)<br />
<br />
== Folder Structure ==<br />
<br />
The following install root folder structure is used by ARK2, based on the default Silex and Composer structure, but adapted to support multi-site and multi-tenant.<br />
<br />
/<br />
|- bin/<br />
|- console<br />
|- build/<br />
|- <see [[ARK2/Frontend|Frontend]]><br />
|- sites/<br />
|- src/<br />
|- tests/<br />
|- var/<br />
|- cache/<br />
|- logs/<br />
|- vendor/<br />
<br />
* The 'bin' folder holds any executable binaries, primarily the admin console script<br />
* The 'build' folder holds tools and source files for building [[ARK2/Frontend|frontends]]<br />
* The 'sites' folder holds the site specific files in a way that supports multi-site installs<br />
* The 'src' folder holds the ARK source code<br />
* The 'tests' folder holds the ARK auto-tests<br />
* The 'var' folder holds transitory site files, such as caches and logs<br />
* The 'vendor' holds the component library code managed by Composer<br />
<br />
Tarball packaging for release will not include the 'build' or 'test' folders, nor the contents of the 'vendor' folder.<br />
<br />
The ARK2 'src' folder is organised as follows:<br />
<br />
/<br />
|- src/<br />
|- ARK/<br />
|- server/<br />
|- php/<br />
|- schema/<br />
|- database/<br />
|- json/<br />
|- web/<br />
|- <see [[ARK2/Frontend|Frontend]]><br />
|- <project>/<br />
<br />
* The 'src/ARK' folder holds the ARK Project specific code, this code is not intended for custom site code and will be over-written by any ARK update<br />
* The 'src/ARK/server' folder holds the ARK Server code, i.e. the backend code, API, database, etc<br />
* The 'src/ARK/web' folder holds the ARK Web code, i.e. the frontend code. See [[ARK2/Frontend|Frontend]] for more details.<br />
* The 'src/<project>' folder would hold custom frontend code where another project wishes to run their own multi-site install with a single custom frontend, see [[ARK2/Frontend|Frontend]] for more details<br />
<br />
The ARK2 'sites' folder is organised as follows:<br />
<br />
/<br />
|- sites/<br />
|- <site>/<br />
|- config/<br />
|- database.json<br />
|- site.json<br />
|- files<br />
|- <see [[ARK2/Files|Files]]><br />
|- src/<br />
|- <see [[ARK2/Frontend|Frontend]]><br />
|- schema/<br />
|- web/<br />
|- .htaccess<br />
|- index.php<br />
|- assets/<br />
|- <see [[ARK2/Frontend|Frontend]]><br />
<br />
* The 'sites/<site>' folder holds all the custom files required for a site. This allows for multi-site installs, as well as a simple site backup/transfer strategy. The <site> name is recommended to be the reverse domain name for the site if unique (e.g. com.example.www) or the subfolder if using a shared domain (e.g. mysite).<br />
* The 'sites/<site>/config' folder holds all the configuration files for the site.<br />
* The 'sites/<site>/files' folder holds all the data files for the site, such as photos and documents. See [[ARK2/Files|Files]] for more details.<br />
* The 'sites/<site>/src' folder holds any custom frontend code, see [[ARK2/Frontend|Frontend]] for more details.<br />
* The 'sites/<site>/schema' folder holds the site specific JSON Schema files.<br />
* The 'sites/<site>/web' folder is the webroot for the site, isolating only the front-controller and assets in this folder improves security.<br />
* The 'sites/<site>/web/assets' folder holds the assets for the frontend(s), usually as a symlink back to the 'src/ARK/web/<frontend>/assets' folder to allow for easy updates. See [[ARK2/Frontend|Frontend]] for more details.<br />
<br />
== Environment ==<br />
<br />
To develop ARK requires the following tools to be installed:<br />
* [https://git-scm.com/ Git]<br />
* [https://getcomposer.org/ Composer]<br />
* [https://nodejs.org/en/ Node.js] (for frontend/theme development only)<br />
<br />
You will also require PHP, a web server (Apache/PHP), a database server (MySQL/PostreSQL/SQLite), a web browser (Chrome/Firefox), and a an API client (Postman).<br />
<br />
The following tools are recommend to adhere to ARK code quality standards:<br />
* php-cs-fixer<br />
* PHP CodeSniffer<br />
* PHP MD<br />
<br />
=== macOS ===<br />
<br />
On macOS, while you can install the requirements via standalone packages, we recommend using HomeBrew as it makes installing and updating all the tools used easier.<br />
<br />
* Install XCode<br />
* Install the Command Line Tools (includes Git)<br />
* Download and install [http://brew.sh/ HomeBrew]<br />
* <pre>brew install node tidy-html5 homebrew/php/composer homebrew/completions/composer-completion homebrew/php/php-cs-fixer homebrew/php/php-code-sniffer homebrew/php/phpmd homebrew/php/sqlformat homebrew/php/phplint</pre><br />
* <pre>npm install -g bootlint csscomb csslint eslint js-beautify jsonlint prettydiff remark-lint sass-lint tidy-markdown</pre><br />
* If ypou need to do front end development you will also need <pre> npm install gulp-sass gulp-sourcemaps gulp-autoprefixer gulp-uglifyjs gulp-rename merge-stream gulp-concat bootstrap-sass</pre><br />
* (Optional) Modify your .profile to enable command-line auto-complete<br />
export HOMEBREW_GITHUB_API_TOKEN="token"<br />
source /Library/Developer/CommandLineTools/usr/share/git-core/git-completion.bash<br />
source /Library/Developer/CommandLineTools/usr/share/git-core/git-prompt.sh<br />
GIT_PS1_SHOWDIRTYSTATE=true<br />
PS1='\u@\h \W$(__git_ps1 " (%s)") \$ '<br />
export PATH=/Users/odysseus/.bin:$PATH<br />
<br />
NOTE add Homebrew to path!!!<br />
<br />
The easiest way to run a webserver and database server is using [https://www.mamp.info/en/ MAMP]. Simply install MAMP and create a soft-link from the MAMP webroot to the location of your ARK git repository. The alternative is to use HomeBrew to configure and run your own install. The HomeBrew method is recommended for developing with PostgreSQL (TODO).<br />
* <pre>brew install homebrew/php/php70 homebrew/php/php70-tidy homebrew/php/php70-xdebug</pre><br />
<br />
<br />
<br />
For QGIS Python:<br />
* 'brew install python qt pyqt'<br />
* 'pip install pep8 autopep8 jedi pb_tool '<br />
<br />
TODO<br />
* xdebug setup<br />
* apache / php /mysql / mamp<br />
* linter/fixer rulesets<br />
<br />
=== Atom Editor ===<br />
<br />
While text editors and IDEs are a deeply personal choice, we recommend using [https://atom.io/ Atom] as it is a cross-platform Open Source editor with powerful plugins to support the tools used by ARK. Using Atom ensures you have an environment consistent with the core ARK developers and ARK development standards.<br />
<br />
After installing Atom, check the commandline tools have been installed.<br />
<br />
Recommended Atom Packages:<br />
* 'apm install atom-autocomplete-php atom-beautify atom-bootstrap3 atom-symfony2 autocomplete-sass doctrine linter-bootlint linter-csslint linter-eslint linter-jsonlint linter-markdown linter-php linter-phpcs linter-phpmd linter-sass-lint linter-tidy linter-twig php-composer-completion php-cs-fixer php-debug php-twig symfony-snippets'<br />
<br />
Useful Atom Packages:<br />
* sublime-style-column-selection<br />
* autoclose-html<br />
* browser-plus<br />
* docblockr<br />
* git-plus<br />
* highlight-line<br />
* highlight-selected<br />
* minimap<br />
* open-recent<br />
* pigments<br />
* platformio-ide-terminal<br />
* project-manager<br />
* sort-lines<br />
* sync-settings<br />
<br />
== Install ==<br />
<br />
composer install</div>Mikehttps://ark.lparchaeology.com/wiki/index.php?title=ARK2/Technical&diff=124650ARK2/Technical2016-12-19T15:48:06Z<p>Mike: /* macOS */</p>
<hr />
<div>= Technical Details =<br />
<br />
== Platforms ==<br />
<br />
* HTML5 will be used<br />
* Browser support restricted to those supported by Bootstrap 3<br />
* PHP: A minimum of v5.6 will be supported (5.6 is in Security Support, 5.7 in active support, see http://php.net/supported-versions.php), v7 will be supported.<br />
* MySQL/MariaDB v5.5 or later (lowest supported MySQL) (5.6 for full-text index, 5.7.5 for spatial index)<br />
* PostgreSQL v9.2 or later<br />
* SQLite 3.7.11 or later (required for multiple inserts)<br />
* mod_rewrite will be required<br />
* PHP intl extension and ICU will be required<br />
<br />
== Standards ==<br />
<br />
The [http://www.php-fig.org/psr/ PHP-FIG standards] will be used:<br />
* PSR-1 and PSR-2 Coding Standards<br />
* PSR-3 Logging Interface for interchangeable logging objects<br />
* PSR-4 Auto-Loading Standard<br />
* PSR-5 PHPDoc Standard (proposed)<br />
* Symfony HTTP Foundation for interchangeable Request/Response objects<br />
* PSR-7 HTTP Message Interface for interchangeable Request/Response objects if required for some components<br />
* All files will be UTF8 using UNIX LF<br />
<br />
Plugins are available for Eclipse, Atom, and other IDEs to automatically check and/or apply these standards.<br />
<br />
== Framework and Components ==<br />
<br />
The chosen framework is the [http://silex.sensiolabs.org/ Silex] micro-framework built using [http://symfony.com/ Symphony] components<br />
<br />
Components will be carefully chosen to be well supported, stable, and interchangeable wherever possible. A number of criteria will be applied in selection:<br />
* Must be standards compliant<br />
* Must be well supported with a solid development history<br />
* Must be well documented<br />
* Must be widely used and supported<br />
* Must have a strong community, small one-person efforts will only be considered if they are the de-facto standard<br />
* Any database access must use Doctrine DBAL or ORM<br />
<br />
Significant and reliable sources of components include:<br />
* [https://packagist.org/ Packagist], the Composer index <br />
* [http://symfony.com/components Symfony]<br />
* [http://thephpleague.com/#packages PHP League]<br />
* [https://zendframework.github.io/ Zend]<br />
* [http://docs.sylius.org/en/latest/components/index.html Sylius], components from an e-commerce platform based on Symfony<br />
* [http://auraphp.com/ Aura]<br />
* [https://github.com/silexphp/Silex/wiki/Third-Party-ServiceProviders-for-Silex-2.x Silex providers]<br />
<br />
* [http://www.userfrosting.com/ User frosting], a UserCake fork with RBAC user management, using Slim2, SBAdmin2, use for ideas<br />
<br />
== PHP Autoloading ==<br />
<br />
PSR-4 is used for packaging, namespace and auto-loading of OO code.<br />
* [https://getcomposer.org/ Composer] will be required for dependency management and PSR-4 auto-loading<br />
* All external libraries will be installed by Composer under vendor/<br />
* All OO classes will be namespaced under ARK\<br />
* All OO code will be under src/<br />
<br />
A good series of articles explaining PSR-4 and modern development and packaging in general can be found at the following:<br />
* http://culttt.com/2013/01/07/what-is-php-composer/<br />
* http://culttt.com/2014/03/12/build-php-package/<br />
* http://culttt.com/2014/05/07/create-psr-4-php-package/<br />
<br />
== Git / Github ==<br />
<br />
Development is hosted on [https://github.com/lparchaeology Github], so you will need a free Github account to contribute code.<br />
<br />
If you are new to Git, you may find the [https://desktop.github.com/ Github desktop application] easier to use. Alternatives are [https://www.sourcetreeapp.com/ SourceTree] and [https://www.gitkraken.com/ GitKraken].<br />
<br />
We will use a simplifed version of the [https://wiki.qt.io/Branch_Guidelines Qt git workflow].<br />
* '''dev''' branch is the semi-stable development branch for the next release<br />
* New feature or bugfix branches are branched off '''dev''' with the name of the author/group/client and the feature or bug number, e.g. '''jlayt/linked-data'''<br />
* Once a new feature is finished and passes testing, it is rebased onto the current '''dev''' and merged via a Github merge request with review<br />
* Alpha and Beta testing releases will be tagged on '''dev''' and not branched<br />
* Once '''dev''' is deemed feature complete and stable enough for a Release Candidate then a release branch will be branched off '''dev''' with the minor release number, e.g. '''release/2.0''', '''release/2.1''', etc<br />
* Further testing will take place on the release branch, with fixes applied to to the release branch as required<br />
* Once ready for release, the release will be tagged with the point release number (e.g. 2.0.0) and all bugfixes merged back into '''dev'''<br />
* Any bugfixes for the stable release will be made in the earliest release branch they apply to and then merged forward through each release then eventually into '''dev''' (merges may be immediate or periodic depending on the volume)<br />
<br />
== Folder Structure ==<br />
<br />
The following install root folder structure is used by ARK2, based on the default Silex and Composer structure, but adapted to support multi-site and multi-tenant.<br />
<br />
/<br />
|- bin/<br />
|- console<br />
|- build/<br />
|- <see [[ARK2/Frontend|Frontend]]><br />
|- sites/<br />
|- src/<br />
|- tests/<br />
|- var/<br />
|- cache/<br />
|- logs/<br />
|- vendor/<br />
<br />
* The 'bin' folder holds any executable binaries, primarily the admin console script<br />
* The 'build' folder holds tools and source files for building [[ARK2/Frontend|frontends]]<br />
* The 'sites' folder holds the site specific files in a way that supports multi-site installs<br />
* The 'src' folder holds the ARK source code<br />
* The 'tests' folder holds the ARK auto-tests<br />
* The 'var' folder holds transitory site files, such as caches and logs<br />
* The 'vendor' holds the component library code managed by Composer<br />
<br />
Tarball packaging for release will not include the 'build' or 'test' folders, nor the contents of the 'vendor' folder.<br />
<br />
The ARK2 'src' folder is organised as follows:<br />
<br />
/<br />
|- src/<br />
|- ARK/<br />
|- server/<br />
|- php/<br />
|- schema/<br />
|- database/<br />
|- json/<br />
|- web/<br />
|- <see [[ARK2/Frontend|Frontend]]><br />
|- <project>/<br />
<br />
* The 'src/ARK' folder holds the ARK Project specific code, this code is not intended for custom site code and will be over-written by any ARK update<br />
* The 'src/ARK/server' folder holds the ARK Server code, i.e. the backend code, API, database, etc<br />
* The 'src/ARK/web' folder holds the ARK Web code, i.e. the frontend code. See [[ARK2/Frontend|Frontend]] for more details.<br />
* The 'src/<project>' folder would hold custom frontend code where another project wishes to run their own multi-site install with a single custom frontend, see [[ARK2/Frontend|Frontend]] for more details<br />
<br />
The ARK2 'sites' folder is organised as follows:<br />
<br />
/<br />
|- sites/<br />
|- <site>/<br />
|- config/<br />
|- database.json<br />
|- site.json<br />
|- files<br />
|- <see [[ARK2/Files|Files]]><br />
|- src/<br />
|- <see [[ARK2/Frontend|Frontend]]><br />
|- schema/<br />
|- web/<br />
|- .htaccess<br />
|- index.php<br />
|- assets/<br />
|- <see [[ARK2/Frontend|Frontend]]><br />
<br />
* The 'sites/<site>' folder holds all the custom files required for a site. This allows for multi-site installs, as well as a simple site backup/transfer strategy. The <site> name is recommended to be the reverse domain name for the site if unique (e.g. com.example.www) or the subfolder if using a shared domain (e.g. mysite).<br />
* The 'sites/<site>/config' folder holds all the configuration files for the site.<br />
* The 'sites/<site>/files' folder holds all the data files for the site, such as photos and documents. See [[ARK2/Files|Files]] for more details.<br />
* The 'sites/<site>/src' folder holds any custom frontend code, see [[ARK2/Frontend|Frontend]] for more details.<br />
* The 'sites/<site>/schema' folder holds the site specific JSON Schema files.<br />
* The 'sites/<site>/web' folder is the webroot for the site, isolating only the front-controller and assets in this folder improves security.<br />
* The 'sites/<site>/web/assets' folder holds the assets for the frontend(s), usually as a symlink back to the 'src/ARK/web/<frontend>/assets' folder to allow for easy updates. See [[ARK2/Frontend|Frontend]] for more details.<br />
<br />
== Environment ==<br />
<br />
To develop ARK requires the following tools to be installed:<br />
* [https://git-scm.com/ Git]<br />
* [https://getcomposer.org/ Composer]<br />
* [https://nodejs.org/en/ Node.js] (for frontend/theme development only)<br />
<br />
You will also require PHP, a web server (Apache/PHP), a database server (MySQL/PostreSQL/SQLite), a web browser (Chrome/Firefox), and a an API client (Postman).<br />
<br />
The following tools are recommend to adhere to ARK code quality standards:<br />
* php-cs-fixer<br />
* PHP CodeSniffer<br />
* PHP MD<br />
<br />
=== macOS ===<br />
<br />
On macOS, while you can install the requirements via standalone packages, we recommend using HomeBrew as it makes installing and updating all the tools used easier.<br />
<br />
* Install XCode<br />
* Install the Command Line Tools (includes Git)<br />
* Download and install [http://brew.sh/ HomeBrew]<br />
* <pre>brew install node tidy-html5 homebrew/php/composer homebrew/completions/composer-completion homebrew/php/php-cs-fixer homebrew/php/php-code-sniffer homebrew/php/phpmd homebrew/php/sqlformat homebrew/php/phplint</pre><br />
* <pre>npm install -g bootlint csscomb csslint eslint js-beautify jsonlint prettydiff remark-lint sass-lint tidy-markdown</pre><br />
* If ypou need to do front end development you will also need <pre> npm install gulp-sass gulp-sourcemaps gulp-autoprefixer gulp-uglifyjs gulp-rename merge-stream gulp-concat </pre><br />
* (Optional) Modify your .profile to enable command-line auto-complete<br />
export HOMEBREW_GITHUB_API_TOKEN="token"<br />
source /Library/Developer/CommandLineTools/usr/share/git-core/git-completion.bash<br />
source /Library/Developer/CommandLineTools/usr/share/git-core/git-prompt.sh<br />
GIT_PS1_SHOWDIRTYSTATE=true<br />
PS1='\u@\h \W$(__git_ps1 " (%s)") \$ '<br />
export PATH=/Users/odysseus/.bin:$PATH<br />
<br />
NOTE add Homebrew to path!!!<br />
<br />
The easiest way to run a webserver and database server is using [https://www.mamp.info/en/ MAMP]. Simply install MAMP and create a soft-link from the MAMP webroot to the location of your ARK git repository. The alternative is to use HomeBrew to configure and run your own install. The HomeBrew method is recommended for developing with PostgreSQL (TODO).<br />
* <pre>brew install homebrew/php/php70 homebrew/php/php70-tidy homebrew/php/php70-xdebug</pre><br />
<br />
<br />
<br />
For QGIS Python:<br />
* 'brew install python qt pyqt'<br />
* 'pip install pep8 autopep8 jedi pb_tool '<br />
<br />
TODO<br />
* xdebug setup<br />
* apache / php /mysql / mamp<br />
* linter/fixer rulesets<br />
<br />
=== Atom Editor ===<br />
<br />
While text editors and IDEs are a deeply personal choice, we recommend using [https://atom.io/ Atom] as it is a cross-platform Open Source editor with powerful plugins to support the tools used by ARK. Using Atom ensures you have an environment consistent with the core ARK developers and ARK development standards.<br />
<br />
After installing Atom, check the commandline tools have been installed.<br />
<br />
Recommended Atom Packages:<br />
* 'apm install atom-autocomplete-php atom-beautify atom-bootstrap3 atom-symfony2 autocomplete-sass doctrine linter-bootlint linter-csslint linter-eslint linter-jsonlint linter-markdown linter-php linter-phpcs linter-phpmd linter-sass-lint linter-tidy linter-twig php-composer-completion php-cs-fixer php-debug php-twig symfony-snippets'<br />
<br />
Useful Atom Packages:<br />
* sublime-style-column-selection<br />
* autoclose-html<br />
* browser-plus<br />
* docblockr<br />
* git-plus<br />
* highlight-line<br />
* highlight-selected<br />
* minimap<br />
* open-recent<br />
* pigments<br />
* platformio-ide-terminal<br />
* project-manager<br />
* sort-lines<br />
* sync-settings<br />
<br />
== Install ==<br />
<br />
composer install</div>Mikehttps://ark.lparchaeology.com/wiki/index.php?title=ARK2/Technical&diff=124648ARK2/Technical2016-12-19T15:44:31Z<p>Mike: /* macOS */</p>
<hr />
<div>= Technical Details =<br />
<br />
== Platforms ==<br />
<br />
* HTML5 will be used<br />
* Browser support restricted to those supported by Bootstrap 3<br />
* PHP: A minimum of v5.6 will be supported (5.6 is in Security Support, 5.7 in active support, see http://php.net/supported-versions.php), v7 will be supported.<br />
* MySQL/MariaDB v5.5 or later (lowest supported MySQL) (5.6 for full-text index, 5.7.5 for spatial index)<br />
* PostgreSQL v9.2 or later<br />
* SQLite 3.7.11 or later (required for multiple inserts)<br />
* mod_rewrite will be required<br />
* PHP intl extension and ICU will be required<br />
<br />
== Standards ==<br />
<br />
The [http://www.php-fig.org/psr/ PHP-FIG standards] will be used:<br />
* PSR-1 and PSR-2 Coding Standards<br />
* PSR-3 Logging Interface for interchangeable logging objects<br />
* PSR-4 Auto-Loading Standard<br />
* PSR-5 PHPDoc Standard (proposed)<br />
* Symfony HTTP Foundation for interchangeable Request/Response objects<br />
* PSR-7 HTTP Message Interface for interchangeable Request/Response objects if required for some components<br />
* All files will be UTF8 using UNIX LF<br />
<br />
Plugins are available for Eclipse, Atom, and other IDEs to automatically check and/or apply these standards.<br />
<br />
== Framework and Components ==<br />
<br />
The chosen framework is the [http://silex.sensiolabs.org/ Silex] micro-framework built using [http://symfony.com/ Symphony] components<br />
<br />
Components will be carefully chosen to be well supported, stable, and interchangeable wherever possible. A number of criteria will be applied in selection:<br />
* Must be standards compliant<br />
* Must be well supported with a solid development history<br />
* Must be well documented<br />
* Must be widely used and supported<br />
* Must have a strong community, small one-person efforts will only be considered if they are the de-facto standard<br />
* Any database access must use Doctrine DBAL or ORM<br />
<br />
Significant and reliable sources of components include:<br />
* [https://packagist.org/ Packagist], the Composer index <br />
* [http://symfony.com/components Symfony]<br />
* [http://thephpleague.com/#packages PHP League]<br />
* [https://zendframework.github.io/ Zend]<br />
* [http://docs.sylius.org/en/latest/components/index.html Sylius], components from an e-commerce platform based on Symfony<br />
* [http://auraphp.com/ Aura]<br />
* [https://github.com/silexphp/Silex/wiki/Third-Party-ServiceProviders-for-Silex-2.x Silex providers]<br />
<br />
* [http://www.userfrosting.com/ User frosting], a UserCake fork with RBAC user management, using Slim2, SBAdmin2, use for ideas<br />
<br />
== PHP Autoloading ==<br />
<br />
PSR-4 is used for packaging, namespace and auto-loading of OO code.<br />
* [https://getcomposer.org/ Composer] will be required for dependency management and PSR-4 auto-loading<br />
* All external libraries will be installed by Composer under vendor/<br />
* All OO classes will be namespaced under ARK\<br />
* All OO code will be under src/<br />
<br />
A good series of articles explaining PSR-4 and modern development and packaging in general can be found at the following:<br />
* http://culttt.com/2013/01/07/what-is-php-composer/<br />
* http://culttt.com/2014/03/12/build-php-package/<br />
* http://culttt.com/2014/05/07/create-psr-4-php-package/<br />
<br />
== Git / Github ==<br />
<br />
Development is hosted on [https://github.com/lparchaeology Github], so you will need a free Github account to contribute code.<br />
<br />
If you are new to Git, you may find the [https://desktop.github.com/ Github desktop application] easier to use. Alternatives are [https://www.sourcetreeapp.com/ SourceTree] and [https://www.gitkraken.com/ GitKraken].<br />
<br />
We will use a simplifed version of the [https://wiki.qt.io/Branch_Guidelines Qt git workflow].<br />
* '''dev''' branch is the semi-stable development branch for the next release<br />
* New feature or bugfix branches are branched off '''dev''' with the name of the author/group/client and the feature or bug number, e.g. '''jlayt/linked-data'''<br />
* Once a new feature is finished and passes testing, it is rebased onto the current '''dev''' and merged via a Github merge request with review<br />
* Alpha and Beta testing releases will be tagged on '''dev''' and not branched<br />
* Once '''dev''' is deemed feature complete and stable enough for a Release Candidate then a release branch will be branched off '''dev''' with the minor release number, e.g. '''release/2.0''', '''release/2.1''', etc<br />
* Further testing will take place on the release branch, with fixes applied to to the release branch as required<br />
* Once ready for release, the release will be tagged with the point release number (e.g. 2.0.0) and all bugfixes merged back into '''dev'''<br />
* Any bugfixes for the stable release will be made in the earliest release branch they apply to and then merged forward through each release then eventually into '''dev''' (merges may be immediate or periodic depending on the volume)<br />
<br />
== Folder Structure ==<br />
<br />
The following install root folder structure is used by ARK2, based on the default Silex and Composer structure, but adapted to support multi-site and multi-tenant.<br />
<br />
/<br />
|- bin/<br />
|- console<br />
|- build/<br />
|- <see [[ARK2/Frontend|Frontend]]><br />
|- sites/<br />
|- src/<br />
|- tests/<br />
|- var/<br />
|- cache/<br />
|- logs/<br />
|- vendor/<br />
<br />
* The 'bin' folder holds any executable binaries, primarily the admin console script<br />
* The 'build' folder holds tools and source files for building [[ARK2/Frontend|frontends]]<br />
* The 'sites' folder holds the site specific files in a way that supports multi-site installs<br />
* The 'src' folder holds the ARK source code<br />
* The 'tests' folder holds the ARK auto-tests<br />
* The 'var' folder holds transitory site files, such as caches and logs<br />
* The 'vendor' holds the component library code managed by Composer<br />
<br />
Tarball packaging for release will not include the 'build' or 'test' folders, nor the contents of the 'vendor' folder.<br />
<br />
The ARK2 'src' folder is organised as follows:<br />
<br />
/<br />
|- src/<br />
|- ARK/<br />
|- server/<br />
|- php/<br />
|- schema/<br />
|- database/<br />
|- json/<br />
|- web/<br />
|- <see [[ARK2/Frontend|Frontend]]><br />
|- <project>/<br />
<br />
* The 'src/ARK' folder holds the ARK Project specific code, this code is not intended for custom site code and will be over-written by any ARK update<br />
* The 'src/ARK/server' folder holds the ARK Server code, i.e. the backend code, API, database, etc<br />
* The 'src/ARK/web' folder holds the ARK Web code, i.e. the frontend code. See [[ARK2/Frontend|Frontend]] for more details.<br />
* The 'src/<project>' folder would hold custom frontend code where another project wishes to run their own multi-site install with a single custom frontend, see [[ARK2/Frontend|Frontend]] for more details<br />
<br />
The ARK2 'sites' folder is organised as follows:<br />
<br />
/<br />
|- sites/<br />
|- <site>/<br />
|- config/<br />
|- database.json<br />
|- site.json<br />
|- files<br />
|- <see [[ARK2/Files|Files]]><br />
|- src/<br />
|- <see [[ARK2/Frontend|Frontend]]><br />
|- schema/<br />
|- web/<br />
|- .htaccess<br />
|- index.php<br />
|- assets/<br />
|- <see [[ARK2/Frontend|Frontend]]><br />
<br />
* The 'sites/<site>' folder holds all the custom files required for a site. This allows for multi-site installs, as well as a simple site backup/transfer strategy. The <site> name is recommended to be the reverse domain name for the site if unique (e.g. com.example.www) or the subfolder if using a shared domain (e.g. mysite).<br />
* The 'sites/<site>/config' folder holds all the configuration files for the site.<br />
* The 'sites/<site>/files' folder holds all the data files for the site, such as photos and documents. See [[ARK2/Files|Files]] for more details.<br />
* The 'sites/<site>/src' folder holds any custom frontend code, see [[ARK2/Frontend|Frontend]] for more details.<br />
* The 'sites/<site>/schema' folder holds the site specific JSON Schema files.<br />
* The 'sites/<site>/web' folder is the webroot for the site, isolating only the front-controller and assets in this folder improves security.<br />
* The 'sites/<site>/web/assets' folder holds the assets for the frontend(s), usually as a symlink back to the 'src/ARK/web/<frontend>/assets' folder to allow for easy updates. See [[ARK2/Frontend|Frontend]] for more details.<br />
<br />
== Environment ==<br />
<br />
To develop ARK requires the following tools to be installed:<br />
* [https://git-scm.com/ Git]<br />
* [https://getcomposer.org/ Composer]<br />
* [https://nodejs.org/en/ Node.js] (for frontend/theme development only)<br />
<br />
You will also require PHP, a web server (Apache/PHP), a database server (MySQL/PostreSQL/SQLite), a web browser (Chrome/Firefox), and a an API client (Postman).<br />
<br />
The following tools are recommend to adhere to ARK code quality standards:<br />
* php-cs-fixer<br />
* PHP CodeSniffer<br />
* PHP MD<br />
<br />
=== macOS ===<br />
<br />
On macOS, while you can install the requirements via standalone packages, we recommend using HomeBrew as it makes installing and updating all the tools used easier.<br />
<br />
* Install XCode<br />
* Install the Command Line Tools (includes Git)<br />
* Download and install [http://brew.sh/ HomeBrew]<br />
* <pre>brew install node tidy-html5 homebrew/php/composer homebrew/completions/composer-completion homebrew/php/php-cs-fixer homebrew/php/php-code-sniffer homebrew/php/phpmd homebrew/php/sqlformat homebrew/php/phplint</pre><br />
* <pre>npm install -g bootlint csscomb csslint eslint js-beautify jsonlint prettydiff remark-lint sass-lint tidy-markdown</pre><br />
* If ypou need to do front end development you will also need <pre> npm install gulp-sass gulp-sourcemaps gulp-autoprefixer gulp-uglifyjs gulp-rename </pre><br />
* (Optional) Modify your .profile to enable command-line auto-complete<br />
export HOMEBREW_GITHUB_API_TOKEN="token"<br />
source /Library/Developer/CommandLineTools/usr/share/git-core/git-completion.bash<br />
source /Library/Developer/CommandLineTools/usr/share/git-core/git-prompt.sh<br />
GIT_PS1_SHOWDIRTYSTATE=true<br />
PS1='\u@\h \W$(__git_ps1 " (%s)") \$ '<br />
export PATH=/Users/odysseus/.bin:$PATH<br />
<br />
NOTE add Homebrew to path!!!<br />
<br />
The easiest way to run a webserver and database server is using [https://www.mamp.info/en/ MAMP]. Simply install MAMP and create a soft-link from the MAMP webroot to the location of your ARK git repository. The alternative is to use HomeBrew to configure and run your own install. The HomeBrew method is recommended for developing with PostgreSQL (TODO).<br />
* <pre>brew install homebrew/php/php70 homebrew/php/php70-tidy homebrew/php/php70-xdebug</pre><br />
<br />
<br />
<br />
For QGIS Python:<br />
* 'brew install python qt pyqt'<br />
* 'pip install pep8 autopep8 jedi pb_tool '<br />
<br />
TODO<br />
* xdebug setup<br />
* apache / php /mysql / mamp<br />
* linter/fixer rulesets<br />
<br />
=== Atom Editor ===<br />
<br />
While text editors and IDEs are a deeply personal choice, we recommend using [https://atom.io/ Atom] as it is a cross-platform Open Source editor with powerful plugins to support the tools used by ARK. Using Atom ensures you have an environment consistent with the core ARK developers and ARK development standards.<br />
<br />
After installing Atom, check the commandline tools have been installed.<br />
<br />
Recommended Atom Packages:<br />
* 'apm install atom-autocomplete-php atom-beautify atom-bootstrap3 atom-symfony2 autocomplete-sass doctrine linter-bootlint linter-csslint linter-eslint linter-jsonlint linter-markdown linter-php linter-phpcs linter-phpmd linter-sass-lint linter-tidy linter-twig php-composer-completion php-cs-fixer php-debug php-twig symfony-snippets'<br />
<br />
Useful Atom Packages:<br />
* sublime-style-column-selection<br />
* autoclose-html<br />
* browser-plus<br />
* docblockr<br />
* git-plus<br />
* highlight-line<br />
* highlight-selected<br />
* minimap<br />
* open-recent<br />
* pigments<br />
* platformio-ide-terminal<br />
* project-manager<br />
* sort-lines<br />
* sync-settings<br />
<br />
== Install ==<br />
<br />
composer install</div>Mikehttps://ark.lparchaeology.com/wiki/index.php?title=ARK2/Technical&diff=124647ARK2/Technical2016-12-19T15:43:56Z<p>Mike: /* macOS */</p>
<hr />
<div>= Technical Details =<br />
<br />
== Platforms ==<br />
<br />
* HTML5 will be used<br />
* Browser support restricted to those supported by Bootstrap 3<br />
* PHP: A minimum of v5.6 will be supported (5.6 is in Security Support, 5.7 in active support, see http://php.net/supported-versions.php), v7 will be supported.<br />
* MySQL/MariaDB v5.5 or later (lowest supported MySQL) (5.6 for full-text index, 5.7.5 for spatial index)<br />
* PostgreSQL v9.2 or later<br />
* SQLite 3.7.11 or later (required for multiple inserts)<br />
* mod_rewrite will be required<br />
* PHP intl extension and ICU will be required<br />
<br />
== Standards ==<br />
<br />
The [http://www.php-fig.org/psr/ PHP-FIG standards] will be used:<br />
* PSR-1 and PSR-2 Coding Standards<br />
* PSR-3 Logging Interface for interchangeable logging objects<br />
* PSR-4 Auto-Loading Standard<br />
* PSR-5 PHPDoc Standard (proposed)<br />
* Symfony HTTP Foundation for interchangeable Request/Response objects<br />
* PSR-7 HTTP Message Interface for interchangeable Request/Response objects if required for some components<br />
* All files will be UTF8 using UNIX LF<br />
<br />
Plugins are available for Eclipse, Atom, and other IDEs to automatically check and/or apply these standards.<br />
<br />
== Framework and Components ==<br />
<br />
The chosen framework is the [http://silex.sensiolabs.org/ Silex] micro-framework built using [http://symfony.com/ Symphony] components<br />
<br />
Components will be carefully chosen to be well supported, stable, and interchangeable wherever possible. A number of criteria will be applied in selection:<br />
* Must be standards compliant<br />
* Must be well supported with a solid development history<br />
* Must be well documented<br />
* Must be widely used and supported<br />
* Must have a strong community, small one-person efforts will only be considered if they are the de-facto standard<br />
* Any database access must use Doctrine DBAL or ORM<br />
<br />
Significant and reliable sources of components include:<br />
* [https://packagist.org/ Packagist], the Composer index <br />
* [http://symfony.com/components Symfony]<br />
* [http://thephpleague.com/#packages PHP League]<br />
* [https://zendframework.github.io/ Zend]<br />
* [http://docs.sylius.org/en/latest/components/index.html Sylius], components from an e-commerce platform based on Symfony<br />
* [http://auraphp.com/ Aura]<br />
* [https://github.com/silexphp/Silex/wiki/Third-Party-ServiceProviders-for-Silex-2.x Silex providers]<br />
<br />
* [http://www.userfrosting.com/ User frosting], a UserCake fork with RBAC user management, using Slim2, SBAdmin2, use for ideas<br />
<br />
== PHP Autoloading ==<br />
<br />
PSR-4 is used for packaging, namespace and auto-loading of OO code.<br />
* [https://getcomposer.org/ Composer] will be required for dependency management and PSR-4 auto-loading<br />
* All external libraries will be installed by Composer under vendor/<br />
* All OO classes will be namespaced under ARK\<br />
* All OO code will be under src/<br />
<br />
A good series of articles explaining PSR-4 and modern development and packaging in general can be found at the following:<br />
* http://culttt.com/2013/01/07/what-is-php-composer/<br />
* http://culttt.com/2014/03/12/build-php-package/<br />
* http://culttt.com/2014/05/07/create-psr-4-php-package/<br />
<br />
== Git / Github ==<br />
<br />
Development is hosted on [https://github.com/lparchaeology Github], so you will need a free Github account to contribute code.<br />
<br />
If you are new to Git, you may find the [https://desktop.github.com/ Github desktop application] easier to use. Alternatives are [https://www.sourcetreeapp.com/ SourceTree] and [https://www.gitkraken.com/ GitKraken].<br />
<br />
We will use a simplifed version of the [https://wiki.qt.io/Branch_Guidelines Qt git workflow].<br />
* '''dev''' branch is the semi-stable development branch for the next release<br />
* New feature or bugfix branches are branched off '''dev''' with the name of the author/group/client and the feature or bug number, e.g. '''jlayt/linked-data'''<br />
* Once a new feature is finished and passes testing, it is rebased onto the current '''dev''' and merged via a Github merge request with review<br />
* Alpha and Beta testing releases will be tagged on '''dev''' and not branched<br />
* Once '''dev''' is deemed feature complete and stable enough for a Release Candidate then a release branch will be branched off '''dev''' with the minor release number, e.g. '''release/2.0''', '''release/2.1''', etc<br />
* Further testing will take place on the release branch, with fixes applied to to the release branch as required<br />
* Once ready for release, the release will be tagged with the point release number (e.g. 2.0.0) and all bugfixes merged back into '''dev'''<br />
* Any bugfixes for the stable release will be made in the earliest release branch they apply to and then merged forward through each release then eventually into '''dev''' (merges may be immediate or periodic depending on the volume)<br />
<br />
== Folder Structure ==<br />
<br />
The following install root folder structure is used by ARK2, based on the default Silex and Composer structure, but adapted to support multi-site and multi-tenant.<br />
<br />
/<br />
|- bin/<br />
|- console<br />
|- build/<br />
|- <see [[ARK2/Frontend|Frontend]]><br />
|- sites/<br />
|- src/<br />
|- tests/<br />
|- var/<br />
|- cache/<br />
|- logs/<br />
|- vendor/<br />
<br />
* The 'bin' folder holds any executable binaries, primarily the admin console script<br />
* The 'build' folder holds tools and source files for building [[ARK2/Frontend|frontends]]<br />
* The 'sites' folder holds the site specific files in a way that supports multi-site installs<br />
* The 'src' folder holds the ARK source code<br />
* The 'tests' folder holds the ARK auto-tests<br />
* The 'var' folder holds transitory site files, such as caches and logs<br />
* The 'vendor' holds the component library code managed by Composer<br />
<br />
Tarball packaging for release will not include the 'build' or 'test' folders, nor the contents of the 'vendor' folder.<br />
<br />
The ARK2 'src' folder is organised as follows:<br />
<br />
/<br />
|- src/<br />
|- ARK/<br />
|- server/<br />
|- php/<br />
|- schema/<br />
|- database/<br />
|- json/<br />
|- web/<br />
|- <see [[ARK2/Frontend|Frontend]]><br />
|- <project>/<br />
<br />
* The 'src/ARK' folder holds the ARK Project specific code, this code is not intended for custom site code and will be over-written by any ARK update<br />
* The 'src/ARK/server' folder holds the ARK Server code, i.e. the backend code, API, database, etc<br />
* The 'src/ARK/web' folder holds the ARK Web code, i.e. the frontend code. See [[ARK2/Frontend|Frontend]] for more details.<br />
* The 'src/<project>' folder would hold custom frontend code where another project wishes to run their own multi-site install with a single custom frontend, see [[ARK2/Frontend|Frontend]] for more details<br />
<br />
The ARK2 'sites' folder is organised as follows:<br />
<br />
/<br />
|- sites/<br />
|- <site>/<br />
|- config/<br />
|- database.json<br />
|- site.json<br />
|- files<br />
|- <see [[ARK2/Files|Files]]><br />
|- src/<br />
|- <see [[ARK2/Frontend|Frontend]]><br />
|- schema/<br />
|- web/<br />
|- .htaccess<br />
|- index.php<br />
|- assets/<br />
|- <see [[ARK2/Frontend|Frontend]]><br />
<br />
* The 'sites/<site>' folder holds all the custom files required for a site. This allows for multi-site installs, as well as a simple site backup/transfer strategy. The <site> name is recommended to be the reverse domain name for the site if unique (e.g. com.example.www) or the subfolder if using a shared domain (e.g. mysite).<br />
* The 'sites/<site>/config' folder holds all the configuration files for the site.<br />
* The 'sites/<site>/files' folder holds all the data files for the site, such as photos and documents. See [[ARK2/Files|Files]] for more details.<br />
* The 'sites/<site>/src' folder holds any custom frontend code, see [[ARK2/Frontend|Frontend]] for more details.<br />
* The 'sites/<site>/schema' folder holds the site specific JSON Schema files.<br />
* The 'sites/<site>/web' folder is the webroot for the site, isolating only the front-controller and assets in this folder improves security.<br />
* The 'sites/<site>/web/assets' folder holds the assets for the frontend(s), usually as a symlink back to the 'src/ARK/web/<frontend>/assets' folder to allow for easy updates. See [[ARK2/Frontend|Frontend]] for more details.<br />
<br />
== Environment ==<br />
<br />
To develop ARK requires the following tools to be installed:<br />
* [https://git-scm.com/ Git]<br />
* [https://getcomposer.org/ Composer]<br />
* [https://nodejs.org/en/ Node.js] (for frontend/theme development only)<br />
<br />
You will also require PHP, a web server (Apache/PHP), a database server (MySQL/PostreSQL/SQLite), a web browser (Chrome/Firefox), and a an API client (Postman).<br />
<br />
The following tools are recommend to adhere to ARK code quality standards:<br />
* php-cs-fixer<br />
* PHP CodeSniffer<br />
* PHP MD<br />
<br />
=== macOS ===<br />
<br />
On macOS, while you can install the requirements via standalone packages, we recommend using HomeBrew as it makes installing and updating all the tools used easier.<br />
<br />
* Install XCode<br />
* Install the Command Line Tools (includes Git)<br />
* Download and install [http://brew.sh/ HomeBrew]<br />
* <pre>brew install node tidy-html5 homebrew/php/composer homebrew/completions/composer-completion homebrew/php/php-cs-fixer homebrew/php/php-code-sniffer homebrew/php/phpmd homebrew/php/sqlformat homebrew/php/phplint</pre><br />
* <pre>npm install -g bootlint csscomb csslint eslint js-beautify jsonlint prettydiff remark-lint sass-lint tidy-markdown</pre><br />
* If ypou need to do front end development you will also need <pre> npm install gulp-sass gulp-sourcemaps gulp-autoprefixer gulp-uglifyjs</pre><br />
* (Optional) Modify your .profile to enable command-line auto-complete<br />
export HOMEBREW_GITHUB_API_TOKEN="token"<br />
source /Library/Developer/CommandLineTools/usr/share/git-core/git-completion.bash<br />
source /Library/Developer/CommandLineTools/usr/share/git-core/git-prompt.sh<br />
GIT_PS1_SHOWDIRTYSTATE=true<br />
PS1='\u@\h \W$(__git_ps1 " (%s)") \$ '<br />
export PATH=/Users/odysseus/.bin:$PATH<br />
<br />
NOTE add Homebrew to path!!!<br />
<br />
The easiest way to run a webserver and database server is using [https://www.mamp.info/en/ MAMP]. Simply install MAMP and create a soft-link from the MAMP webroot to the location of your ARK git repository. The alternative is to use HomeBrew to configure and run your own install. The HomeBrew method is recommended for developing with PostgreSQL (TODO).<br />
* <pre>brew install homebrew/php/php70 homebrew/php/php70-tidy homebrew/php/php70-xdebug</pre><br />
<br />
<br />
<br />
For QGIS Python:<br />
* 'brew install python qt pyqt'<br />
* 'pip install pep8 autopep8 jedi pb_tool '<br />
<br />
TODO<br />
* xdebug setup<br />
* apache / php /mysql / mamp<br />
* linter/fixer rulesets<br />
<br />
=== Atom Editor ===<br />
<br />
While text editors and IDEs are a deeply personal choice, we recommend using [https://atom.io/ Atom] as it is a cross-platform Open Source editor with powerful plugins to support the tools used by ARK. Using Atom ensures you have an environment consistent with the core ARK developers and ARK development standards.<br />
<br />
After installing Atom, check the commandline tools have been installed.<br />
<br />
Recommended Atom Packages:<br />
* 'apm install atom-autocomplete-php atom-beautify atom-bootstrap3 atom-symfony2 autocomplete-sass doctrine linter-bootlint linter-csslint linter-eslint linter-jsonlint linter-markdown linter-php linter-phpcs linter-phpmd linter-sass-lint linter-tidy linter-twig php-composer-completion php-cs-fixer php-debug php-twig symfony-snippets'<br />
<br />
Useful Atom Packages:<br />
* sublime-style-column-selection<br />
* autoclose-html<br />
* browser-plus<br />
* docblockr<br />
* git-plus<br />
* highlight-line<br />
* highlight-selected<br />
* minimap<br />
* open-recent<br />
* pigments<br />
* platformio-ide-terminal<br />
* project-manager<br />
* sort-lines<br />
* sync-settings<br />
<br />
== Install ==<br />
<br />
composer install</div>Mikehttps://ark.lparchaeology.com/wiki/index.php?title=ARK2/Technical&diff=124646ARK2/Technical2016-12-19T15:43:29Z<p>Mike: /* macOS */</p>
<hr />
<div>= Technical Details =<br />
<br />
== Platforms ==<br />
<br />
* HTML5 will be used<br />
* Browser support restricted to those supported by Bootstrap 3<br />
* PHP: A minimum of v5.6 will be supported (5.6 is in Security Support, 5.7 in active support, see http://php.net/supported-versions.php), v7 will be supported.<br />
* MySQL/MariaDB v5.5 or later (lowest supported MySQL) (5.6 for full-text index, 5.7.5 for spatial index)<br />
* PostgreSQL v9.2 or later<br />
* SQLite 3.7.11 or later (required for multiple inserts)<br />
* mod_rewrite will be required<br />
* PHP intl extension and ICU will be required<br />
<br />
== Standards ==<br />
<br />
The [http://www.php-fig.org/psr/ PHP-FIG standards] will be used:<br />
* PSR-1 and PSR-2 Coding Standards<br />
* PSR-3 Logging Interface for interchangeable logging objects<br />
* PSR-4 Auto-Loading Standard<br />
* PSR-5 PHPDoc Standard (proposed)<br />
* Symfony HTTP Foundation for interchangeable Request/Response objects<br />
* PSR-7 HTTP Message Interface for interchangeable Request/Response objects if required for some components<br />
* All files will be UTF8 using UNIX LF<br />
<br />
Plugins are available for Eclipse, Atom, and other IDEs to automatically check and/or apply these standards.<br />
<br />
== Framework and Components ==<br />
<br />
The chosen framework is the [http://silex.sensiolabs.org/ Silex] micro-framework built using [http://symfony.com/ Symphony] components<br />
<br />
Components will be carefully chosen to be well supported, stable, and interchangeable wherever possible. A number of criteria will be applied in selection:<br />
* Must be standards compliant<br />
* Must be well supported with a solid development history<br />
* Must be well documented<br />
* Must be widely used and supported<br />
* Must have a strong community, small one-person efforts will only be considered if they are the de-facto standard<br />
* Any database access must use Doctrine DBAL or ORM<br />
<br />
Significant and reliable sources of components include:<br />
* [https://packagist.org/ Packagist], the Composer index <br />
* [http://symfony.com/components Symfony]<br />
* [http://thephpleague.com/#packages PHP League]<br />
* [https://zendframework.github.io/ Zend]<br />
* [http://docs.sylius.org/en/latest/components/index.html Sylius], components from an e-commerce platform based on Symfony<br />
* [http://auraphp.com/ Aura]<br />
* [https://github.com/silexphp/Silex/wiki/Third-Party-ServiceProviders-for-Silex-2.x Silex providers]<br />
<br />
* [http://www.userfrosting.com/ User frosting], a UserCake fork with RBAC user management, using Slim2, SBAdmin2, use for ideas<br />
<br />
== PHP Autoloading ==<br />
<br />
PSR-4 is used for packaging, namespace and auto-loading of OO code.<br />
* [https://getcomposer.org/ Composer] will be required for dependency management and PSR-4 auto-loading<br />
* All external libraries will be installed by Composer under vendor/<br />
* All OO classes will be namespaced under ARK\<br />
* All OO code will be under src/<br />
<br />
A good series of articles explaining PSR-4 and modern development and packaging in general can be found at the following:<br />
* http://culttt.com/2013/01/07/what-is-php-composer/<br />
* http://culttt.com/2014/03/12/build-php-package/<br />
* http://culttt.com/2014/05/07/create-psr-4-php-package/<br />
<br />
== Git / Github ==<br />
<br />
Development is hosted on [https://github.com/lparchaeology Github], so you will need a free Github account to contribute code.<br />
<br />
If you are new to Git, you may find the [https://desktop.github.com/ Github desktop application] easier to use. Alternatives are [https://www.sourcetreeapp.com/ SourceTree] and [https://www.gitkraken.com/ GitKraken].<br />
<br />
We will use a simplifed version of the [https://wiki.qt.io/Branch_Guidelines Qt git workflow].<br />
* '''dev''' branch is the semi-stable development branch for the next release<br />
* New feature or bugfix branches are branched off '''dev''' with the name of the author/group/client and the feature or bug number, e.g. '''jlayt/linked-data'''<br />
* Once a new feature is finished and passes testing, it is rebased onto the current '''dev''' and merged via a Github merge request with review<br />
* Alpha and Beta testing releases will be tagged on '''dev''' and not branched<br />
* Once '''dev''' is deemed feature complete and stable enough for a Release Candidate then a release branch will be branched off '''dev''' with the minor release number, e.g. '''release/2.0''', '''release/2.1''', etc<br />
* Further testing will take place on the release branch, with fixes applied to to the release branch as required<br />
* Once ready for release, the release will be tagged with the point release number (e.g. 2.0.0) and all bugfixes merged back into '''dev'''<br />
* Any bugfixes for the stable release will be made in the earliest release branch they apply to and then merged forward through each release then eventually into '''dev''' (merges may be immediate or periodic depending on the volume)<br />
<br />
== Folder Structure ==<br />
<br />
The following install root folder structure is used by ARK2, based on the default Silex and Composer structure, but adapted to support multi-site and multi-tenant.<br />
<br />
/<br />
|- bin/<br />
|- console<br />
|- build/<br />
|- <see [[ARK2/Frontend|Frontend]]><br />
|- sites/<br />
|- src/<br />
|- tests/<br />
|- var/<br />
|- cache/<br />
|- logs/<br />
|- vendor/<br />
<br />
* The 'bin' folder holds any executable binaries, primarily the admin console script<br />
* The 'build' folder holds tools and source files for building [[ARK2/Frontend|frontends]]<br />
* The 'sites' folder holds the site specific files in a way that supports multi-site installs<br />
* The 'src' folder holds the ARK source code<br />
* The 'tests' folder holds the ARK auto-tests<br />
* The 'var' folder holds transitory site files, such as caches and logs<br />
* The 'vendor' holds the component library code managed by Composer<br />
<br />
Tarball packaging for release will not include the 'build' or 'test' folders, nor the contents of the 'vendor' folder.<br />
<br />
The ARK2 'src' folder is organised as follows:<br />
<br />
/<br />
|- src/<br />
|- ARK/<br />
|- server/<br />
|- php/<br />
|- schema/<br />
|- database/<br />
|- json/<br />
|- web/<br />
|- <see [[ARK2/Frontend|Frontend]]><br />
|- <project>/<br />
<br />
* The 'src/ARK' folder holds the ARK Project specific code, this code is not intended for custom site code and will be over-written by any ARK update<br />
* The 'src/ARK/server' folder holds the ARK Server code, i.e. the backend code, API, database, etc<br />
* The 'src/ARK/web' folder holds the ARK Web code, i.e. the frontend code. See [[ARK2/Frontend|Frontend]] for more details.<br />
* The 'src/<project>' folder would hold custom frontend code where another project wishes to run their own multi-site install with a single custom frontend, see [[ARK2/Frontend|Frontend]] for more details<br />
<br />
The ARK2 'sites' folder is organised as follows:<br />
<br />
/<br />
|- sites/<br />
|- <site>/<br />
|- config/<br />
|- database.json<br />
|- site.json<br />
|- files<br />
|- <see [[ARK2/Files|Files]]><br />
|- src/<br />
|- <see [[ARK2/Frontend|Frontend]]><br />
|- schema/<br />
|- web/<br />
|- .htaccess<br />
|- index.php<br />
|- assets/<br />
|- <see [[ARK2/Frontend|Frontend]]><br />
<br />
* The 'sites/<site>' folder holds all the custom files required for a site. This allows for multi-site installs, as well as a simple site backup/transfer strategy. The <site> name is recommended to be the reverse domain name for the site if unique (e.g. com.example.www) or the subfolder if using a shared domain (e.g. mysite).<br />
* The 'sites/<site>/config' folder holds all the configuration files for the site.<br />
* The 'sites/<site>/files' folder holds all the data files for the site, such as photos and documents. See [[ARK2/Files|Files]] for more details.<br />
* The 'sites/<site>/src' folder holds any custom frontend code, see [[ARK2/Frontend|Frontend]] for more details.<br />
* The 'sites/<site>/schema' folder holds the site specific JSON Schema files.<br />
* The 'sites/<site>/web' folder is the webroot for the site, isolating only the front-controller and assets in this folder improves security.<br />
* The 'sites/<site>/web/assets' folder holds the assets for the frontend(s), usually as a symlink back to the 'src/ARK/web/<frontend>/assets' folder to allow for easy updates. See [[ARK2/Frontend|Frontend]] for more details.<br />
<br />
== Environment ==<br />
<br />
To develop ARK requires the following tools to be installed:<br />
* [https://git-scm.com/ Git]<br />
* [https://getcomposer.org/ Composer]<br />
* [https://nodejs.org/en/ Node.js] (for frontend/theme development only)<br />
<br />
You will also require PHP, a web server (Apache/PHP), a database server (MySQL/PostreSQL/SQLite), a web browser (Chrome/Firefox), and a an API client (Postman).<br />
<br />
The following tools are recommend to adhere to ARK code quality standards:<br />
* php-cs-fixer<br />
* PHP CodeSniffer<br />
* PHP MD<br />
<br />
=== macOS ===<br />
<br />
On macOS, while you can install the requirements via standalone packages, we recommend using HomeBrew as it makes installing and updating all the tools used easier.<br />
<br />
* Install XCode<br />
* Install the Command Line Tools (includes Git)<br />
* Download and install [http://brew.sh/ HomeBrew]<br />
* <pre>brew install node tidy-html5 homebrew/php/composer homebrew/completions/composer-completion homebrew/php/php-cs-fixer homebrew/php/php-code-sniffer homebrew/php/phpmd homebrew/php/sqlformat homebrew/php/phplint</pre><br />
* <pre>npm install -g bootlint csscomb csslint eslint js-beautify jsonlint prettydiff remark-lint sass-lint tidy-markdown</pre><br />
* If ypou need to do front end development you will also need <pre> npm install gulp-sass gulp-sourcemaps gulp-autoprefixer </pre><br />
* (Optional) Modify your .profile to enable command-line auto-complete<br />
export HOMEBREW_GITHUB_API_TOKEN="token"<br />
source /Library/Developer/CommandLineTools/usr/share/git-core/git-completion.bash<br />
source /Library/Developer/CommandLineTools/usr/share/git-core/git-prompt.sh<br />
GIT_PS1_SHOWDIRTYSTATE=true<br />
PS1='\u@\h \W$(__git_ps1 " (%s)") \$ '<br />
export PATH=/Users/odysseus/.bin:$PATH<br />
<br />
NOTE add Homebrew to path!!!<br />
<br />
The easiest way to run a webserver and database server is using [https://www.mamp.info/en/ MAMP]. Simply install MAMP and create a soft-link from the MAMP webroot to the location of your ARK git repository. The alternative is to use HomeBrew to configure and run your own install. The HomeBrew method is recommended for developing with PostgreSQL (TODO).<br />
* <pre>brew install homebrew/php/php70 homebrew/php/php70-tidy homebrew/php/php70-xdebug</pre><br />
<br />
<br />
<br />
For QGIS Python:<br />
* 'brew install python qt pyqt'<br />
* 'pip install pep8 autopep8 jedi pb_tool '<br />
<br />
TODO<br />
* xdebug setup<br />
* apache / php /mysql / mamp<br />
* linter/fixer rulesets<br />
<br />
=== Atom Editor ===<br />
<br />
While text editors and IDEs are a deeply personal choice, we recommend using [https://atom.io/ Atom] as it is a cross-platform Open Source editor with powerful plugins to support the tools used by ARK. Using Atom ensures you have an environment consistent with the core ARK developers and ARK development standards.<br />
<br />
After installing Atom, check the commandline tools have been installed.<br />
<br />
Recommended Atom Packages:<br />
* 'apm install atom-autocomplete-php atom-beautify atom-bootstrap3 atom-symfony2 autocomplete-sass doctrine linter-bootlint linter-csslint linter-eslint linter-jsonlint linter-markdown linter-php linter-phpcs linter-phpmd linter-sass-lint linter-tidy linter-twig php-composer-completion php-cs-fixer php-debug php-twig symfony-snippets'<br />
<br />
Useful Atom Packages:<br />
* sublime-style-column-selection<br />
* autoclose-html<br />
* browser-plus<br />
* docblockr<br />
* git-plus<br />
* highlight-line<br />
* highlight-selected<br />
* minimap<br />
* open-recent<br />
* pigments<br />
* platformio-ide-terminal<br />
* project-manager<br />
* sort-lines<br />
* sync-settings<br />
<br />
== Install ==<br />
<br />
composer install</div>Mikehttps://ark.lparchaeology.com/wiki/index.php?title=ARK2/Technical&diff=124645ARK2/Technical2016-12-19T15:42:51Z<p>Mike: /* macOS */</p>
<hr />
<div>= Technical Details =<br />
<br />
== Platforms ==<br />
<br />
* HTML5 will be used<br />
* Browser support restricted to those supported by Bootstrap 3<br />
* PHP: A minimum of v5.6 will be supported (5.6 is in Security Support, 5.7 in active support, see http://php.net/supported-versions.php), v7 will be supported.<br />
* MySQL/MariaDB v5.5 or later (lowest supported MySQL) (5.6 for full-text index, 5.7.5 for spatial index)<br />
* PostgreSQL v9.2 or later<br />
* SQLite 3.7.11 or later (required for multiple inserts)<br />
* mod_rewrite will be required<br />
* PHP intl extension and ICU will be required<br />
<br />
== Standards ==<br />
<br />
The [http://www.php-fig.org/psr/ PHP-FIG standards] will be used:<br />
* PSR-1 and PSR-2 Coding Standards<br />
* PSR-3 Logging Interface for interchangeable logging objects<br />
* PSR-4 Auto-Loading Standard<br />
* PSR-5 PHPDoc Standard (proposed)<br />
* Symfony HTTP Foundation for interchangeable Request/Response objects<br />
* PSR-7 HTTP Message Interface for interchangeable Request/Response objects if required for some components<br />
* All files will be UTF8 using UNIX LF<br />
<br />
Plugins are available for Eclipse, Atom, and other IDEs to automatically check and/or apply these standards.<br />
<br />
== Framework and Components ==<br />
<br />
The chosen framework is the [http://silex.sensiolabs.org/ Silex] micro-framework built using [http://symfony.com/ Symphony] components<br />
<br />
Components will be carefully chosen to be well supported, stable, and interchangeable wherever possible. A number of criteria will be applied in selection:<br />
* Must be standards compliant<br />
* Must be well supported with a solid development history<br />
* Must be well documented<br />
* Must be widely used and supported<br />
* Must have a strong community, small one-person efforts will only be considered if they are the de-facto standard<br />
* Any database access must use Doctrine DBAL or ORM<br />
<br />
Significant and reliable sources of components include:<br />
* [https://packagist.org/ Packagist], the Composer index <br />
* [http://symfony.com/components Symfony]<br />
* [http://thephpleague.com/#packages PHP League]<br />
* [https://zendframework.github.io/ Zend]<br />
* [http://docs.sylius.org/en/latest/components/index.html Sylius], components from an e-commerce platform based on Symfony<br />
* [http://auraphp.com/ Aura]<br />
* [https://github.com/silexphp/Silex/wiki/Third-Party-ServiceProviders-for-Silex-2.x Silex providers]<br />
<br />
* [http://www.userfrosting.com/ User frosting], a UserCake fork with RBAC user management, using Slim2, SBAdmin2, use for ideas<br />
<br />
== PHP Autoloading ==<br />
<br />
PSR-4 is used for packaging, namespace and auto-loading of OO code.<br />
* [https://getcomposer.org/ Composer] will be required for dependency management and PSR-4 auto-loading<br />
* All external libraries will be installed by Composer under vendor/<br />
* All OO classes will be namespaced under ARK\<br />
* All OO code will be under src/<br />
<br />
A good series of articles explaining PSR-4 and modern development and packaging in general can be found at the following:<br />
* http://culttt.com/2013/01/07/what-is-php-composer/<br />
* http://culttt.com/2014/03/12/build-php-package/<br />
* http://culttt.com/2014/05/07/create-psr-4-php-package/<br />
<br />
== Git / Github ==<br />
<br />
Development is hosted on [https://github.com/lparchaeology Github], so you will need a free Github account to contribute code.<br />
<br />
If you are new to Git, you may find the [https://desktop.github.com/ Github desktop application] easier to use. Alternatives are [https://www.sourcetreeapp.com/ SourceTree] and [https://www.gitkraken.com/ GitKraken].<br />
<br />
We will use a simplifed version of the [https://wiki.qt.io/Branch_Guidelines Qt git workflow].<br />
* '''dev''' branch is the semi-stable development branch for the next release<br />
* New feature or bugfix branches are branched off '''dev''' with the name of the author/group/client and the feature or bug number, e.g. '''jlayt/linked-data'''<br />
* Once a new feature is finished and passes testing, it is rebased onto the current '''dev''' and merged via a Github merge request with review<br />
* Alpha and Beta testing releases will be tagged on '''dev''' and not branched<br />
* Once '''dev''' is deemed feature complete and stable enough for a Release Candidate then a release branch will be branched off '''dev''' with the minor release number, e.g. '''release/2.0''', '''release/2.1''', etc<br />
* Further testing will take place on the release branch, with fixes applied to to the release branch as required<br />
* Once ready for release, the release will be tagged with the point release number (e.g. 2.0.0) and all bugfixes merged back into '''dev'''<br />
* Any bugfixes for the stable release will be made in the earliest release branch they apply to and then merged forward through each release then eventually into '''dev''' (merges may be immediate or periodic depending on the volume)<br />
<br />
== Folder Structure ==<br />
<br />
The following install root folder structure is used by ARK2, based on the default Silex and Composer structure, but adapted to support multi-site and multi-tenant.<br />
<br />
/<br />
|- bin/<br />
|- console<br />
|- build/<br />
|- <see [[ARK2/Frontend|Frontend]]><br />
|- sites/<br />
|- src/<br />
|- tests/<br />
|- var/<br />
|- cache/<br />
|- logs/<br />
|- vendor/<br />
<br />
* The 'bin' folder holds any executable binaries, primarily the admin console script<br />
* The 'build' folder holds tools and source files for building [[ARK2/Frontend|frontends]]<br />
* The 'sites' folder holds the site specific files in a way that supports multi-site installs<br />
* The 'src' folder holds the ARK source code<br />
* The 'tests' folder holds the ARK auto-tests<br />
* The 'var' folder holds transitory site files, such as caches and logs<br />
* The 'vendor' holds the component library code managed by Composer<br />
<br />
Tarball packaging for release will not include the 'build' or 'test' folders, nor the contents of the 'vendor' folder.<br />
<br />
The ARK2 'src' folder is organised as follows:<br />
<br />
/<br />
|- src/<br />
|- ARK/<br />
|- server/<br />
|- php/<br />
|- schema/<br />
|- database/<br />
|- json/<br />
|- web/<br />
|- <see [[ARK2/Frontend|Frontend]]><br />
|- <project>/<br />
<br />
* The 'src/ARK' folder holds the ARK Project specific code, this code is not intended for custom site code and will be over-written by any ARK update<br />
* The 'src/ARK/server' folder holds the ARK Server code, i.e. the backend code, API, database, etc<br />
* The 'src/ARK/web' folder holds the ARK Web code, i.e. the frontend code. See [[ARK2/Frontend|Frontend]] for more details.<br />
* The 'src/<project>' folder would hold custom frontend code where another project wishes to run their own multi-site install with a single custom frontend, see [[ARK2/Frontend|Frontend]] for more details<br />
<br />
The ARK2 'sites' folder is organised as follows:<br />
<br />
/<br />
|- sites/<br />
|- <site>/<br />
|- config/<br />
|- database.json<br />
|- site.json<br />
|- files<br />
|- <see [[ARK2/Files|Files]]><br />
|- src/<br />
|- <see [[ARK2/Frontend|Frontend]]><br />
|- schema/<br />
|- web/<br />
|- .htaccess<br />
|- index.php<br />
|- assets/<br />
|- <see [[ARK2/Frontend|Frontend]]><br />
<br />
* The 'sites/<site>' folder holds all the custom files required for a site. This allows for multi-site installs, as well as a simple site backup/transfer strategy. The <site> name is recommended to be the reverse domain name for the site if unique (e.g. com.example.www) or the subfolder if using a shared domain (e.g. mysite).<br />
* The 'sites/<site>/config' folder holds all the configuration files for the site.<br />
* The 'sites/<site>/files' folder holds all the data files for the site, such as photos and documents. See [[ARK2/Files|Files]] for more details.<br />
* The 'sites/<site>/src' folder holds any custom frontend code, see [[ARK2/Frontend|Frontend]] for more details.<br />
* The 'sites/<site>/schema' folder holds the site specific JSON Schema files.<br />
* The 'sites/<site>/web' folder is the webroot for the site, isolating only the front-controller and assets in this folder improves security.<br />
* The 'sites/<site>/web/assets' folder holds the assets for the frontend(s), usually as a symlink back to the 'src/ARK/web/<frontend>/assets' folder to allow for easy updates. See [[ARK2/Frontend|Frontend]] for more details.<br />
<br />
== Environment ==<br />
<br />
To develop ARK requires the following tools to be installed:<br />
* [https://git-scm.com/ Git]<br />
* [https://getcomposer.org/ Composer]<br />
* [https://nodejs.org/en/ Node.js] (for frontend/theme development only)<br />
<br />
You will also require PHP, a web server (Apache/PHP), a database server (MySQL/PostreSQL/SQLite), a web browser (Chrome/Firefox), and a an API client (Postman).<br />
<br />
The following tools are recommend to adhere to ARK code quality standards:<br />
* php-cs-fixer<br />
* PHP CodeSniffer<br />
* PHP MD<br />
<br />
=== macOS ===<br />
<br />
On macOS, while you can install the requirements via standalone packages, we recommend using HomeBrew as it makes installing and updating all the tools used easier.<br />
<br />
* Install XCode<br />
* Install the Command Line Tools (includes Git)<br />
* Download and install [http://brew.sh/ HomeBrew]<br />
* <pre>brew install node tidy-html5 homebrew/php/composer homebrew/completions/composer-completion homebrew/php/php-cs-fixer homebrew/php/php-code-sniffer homebrew/php/phpmd homebrew/php/sqlformat homebrew/php/phplint</pre><br />
* <pre>npm install -g bootlint csscomb csslint eslint js-beautify jsonlint prettydiff remark-lint sass-lint tidy-markdown</pre><br />
* If ypou need to do front end development you will also need <pre> npm install gulp-sass gulp-sourcemaps</pre><br />
* (Optional) Modify your .profile to enable command-line auto-complete<br />
export HOMEBREW_GITHUB_API_TOKEN="token"<br />
source /Library/Developer/CommandLineTools/usr/share/git-core/git-completion.bash<br />
source /Library/Developer/CommandLineTools/usr/share/git-core/git-prompt.sh<br />
GIT_PS1_SHOWDIRTYSTATE=true<br />
PS1='\u@\h \W$(__git_ps1 " (%s)") \$ '<br />
export PATH=/Users/odysseus/.bin:$PATH<br />
<br />
NOTE add Homebrew to path!!!<br />
<br />
The easiest way to run a webserver and database server is using [https://www.mamp.info/en/ MAMP]. Simply install MAMP and create a soft-link from the MAMP webroot to the location of your ARK git repository. The alternative is to use HomeBrew to configure and run your own install. The HomeBrew method is recommended for developing with PostgreSQL (TODO).<br />
* <pre>brew install homebrew/php/php70 homebrew/php/php70-tidy homebrew/php/php70-xdebug</pre><br />
<br />
<br />
<br />
For QGIS Python:<br />
* 'brew install python qt pyqt'<br />
* 'pip install pep8 autopep8 jedi pb_tool '<br />
<br />
TODO<br />
* xdebug setup<br />
* apache / php /mysql / mamp<br />
* linter/fixer rulesets<br />
<br />
=== Atom Editor ===<br />
<br />
While text editors and IDEs are a deeply personal choice, we recommend using [https://atom.io/ Atom] as it is a cross-platform Open Source editor with powerful plugins to support the tools used by ARK. Using Atom ensures you have an environment consistent with the core ARK developers and ARK development standards.<br />
<br />
After installing Atom, check the commandline tools have been installed.<br />
<br />
Recommended Atom Packages:<br />
* 'apm install atom-autocomplete-php atom-beautify atom-bootstrap3 atom-symfony2 autocomplete-sass doctrine linter-bootlint linter-csslint linter-eslint linter-jsonlint linter-markdown linter-php linter-phpcs linter-phpmd linter-sass-lint linter-tidy linter-twig php-composer-completion php-cs-fixer php-debug php-twig symfony-snippets'<br />
<br />
Useful Atom Packages:<br />
* sublime-style-column-selection<br />
* autoclose-html<br />
* browser-plus<br />
* docblockr<br />
* git-plus<br />
* highlight-line<br />
* highlight-selected<br />
* minimap<br />
* open-recent<br />
* pigments<br />
* platformio-ide-terminal<br />
* project-manager<br />
* sort-lines<br />
* sync-settings<br />
<br />
== Install ==<br />
<br />
composer install</div>Mikehttps://ark.lparchaeology.com/wiki/index.php?title=ARK2/Technical&diff=124644ARK2/Technical2016-12-19T15:42:27Z<p>Mike: /* macOS */</p>
<hr />
<div>= Technical Details =<br />
<br />
== Platforms ==<br />
<br />
* HTML5 will be used<br />
* Browser support restricted to those supported by Bootstrap 3<br />
* PHP: A minimum of v5.6 will be supported (5.6 is in Security Support, 5.7 in active support, see http://php.net/supported-versions.php), v7 will be supported.<br />
* MySQL/MariaDB v5.5 or later (lowest supported MySQL) (5.6 for full-text index, 5.7.5 for spatial index)<br />
* PostgreSQL v9.2 or later<br />
* SQLite 3.7.11 or later (required for multiple inserts)<br />
* mod_rewrite will be required<br />
* PHP intl extension and ICU will be required<br />
<br />
== Standards ==<br />
<br />
The [http://www.php-fig.org/psr/ PHP-FIG standards] will be used:<br />
* PSR-1 and PSR-2 Coding Standards<br />
* PSR-3 Logging Interface for interchangeable logging objects<br />
* PSR-4 Auto-Loading Standard<br />
* PSR-5 PHPDoc Standard (proposed)<br />
* Symfony HTTP Foundation for interchangeable Request/Response objects<br />
* PSR-7 HTTP Message Interface for interchangeable Request/Response objects if required for some components<br />
* All files will be UTF8 using UNIX LF<br />
<br />
Plugins are available for Eclipse, Atom, and other IDEs to automatically check and/or apply these standards.<br />
<br />
== Framework and Components ==<br />
<br />
The chosen framework is the [http://silex.sensiolabs.org/ Silex] micro-framework built using [http://symfony.com/ Symphony] components<br />
<br />
Components will be carefully chosen to be well supported, stable, and interchangeable wherever possible. A number of criteria will be applied in selection:<br />
* Must be standards compliant<br />
* Must be well supported with a solid development history<br />
* Must be well documented<br />
* Must be widely used and supported<br />
* Must have a strong community, small one-person efforts will only be considered if they are the de-facto standard<br />
* Any database access must use Doctrine DBAL or ORM<br />
<br />
Significant and reliable sources of components include:<br />
* [https://packagist.org/ Packagist], the Composer index <br />
* [http://symfony.com/components Symfony]<br />
* [http://thephpleague.com/#packages PHP League]<br />
* [https://zendframework.github.io/ Zend]<br />
* [http://docs.sylius.org/en/latest/components/index.html Sylius], components from an e-commerce platform based on Symfony<br />
* [http://auraphp.com/ Aura]<br />
* [https://github.com/silexphp/Silex/wiki/Third-Party-ServiceProviders-for-Silex-2.x Silex providers]<br />
<br />
* [http://www.userfrosting.com/ User frosting], a UserCake fork with RBAC user management, using Slim2, SBAdmin2, use for ideas<br />
<br />
== PHP Autoloading ==<br />
<br />
PSR-4 is used for packaging, namespace and auto-loading of OO code.<br />
* [https://getcomposer.org/ Composer] will be required for dependency management and PSR-4 auto-loading<br />
* All external libraries will be installed by Composer under vendor/<br />
* All OO classes will be namespaced under ARK\<br />
* All OO code will be under src/<br />
<br />
A good series of articles explaining PSR-4 and modern development and packaging in general can be found at the following:<br />
* http://culttt.com/2013/01/07/what-is-php-composer/<br />
* http://culttt.com/2014/03/12/build-php-package/<br />
* http://culttt.com/2014/05/07/create-psr-4-php-package/<br />
<br />
== Git / Github ==<br />
<br />
Development is hosted on [https://github.com/lparchaeology Github], so you will need a free Github account to contribute code.<br />
<br />
If you are new to Git, you may find the [https://desktop.github.com/ Github desktop application] easier to use. Alternatives are [https://www.sourcetreeapp.com/ SourceTree] and [https://www.gitkraken.com/ GitKraken].<br />
<br />
We will use a simplifed version of the [https://wiki.qt.io/Branch_Guidelines Qt git workflow].<br />
* '''dev''' branch is the semi-stable development branch for the next release<br />
* New feature or bugfix branches are branched off '''dev''' with the name of the author/group/client and the feature or bug number, e.g. '''jlayt/linked-data'''<br />
* Once a new feature is finished and passes testing, it is rebased onto the current '''dev''' and merged via a Github merge request with review<br />
* Alpha and Beta testing releases will be tagged on '''dev''' and not branched<br />
* Once '''dev''' is deemed feature complete and stable enough for a Release Candidate then a release branch will be branched off '''dev''' with the minor release number, e.g. '''release/2.0''', '''release/2.1''', etc<br />
* Further testing will take place on the release branch, with fixes applied to to the release branch as required<br />
* Once ready for release, the release will be tagged with the point release number (e.g. 2.0.0) and all bugfixes merged back into '''dev'''<br />
* Any bugfixes for the stable release will be made in the earliest release branch they apply to and then merged forward through each release then eventually into '''dev''' (merges may be immediate or periodic depending on the volume)<br />
<br />
== Folder Structure ==<br />
<br />
The following install root folder structure is used by ARK2, based on the default Silex and Composer structure, but adapted to support multi-site and multi-tenant.<br />
<br />
/<br />
|- bin/<br />
|- console<br />
|- build/<br />
|- <see [[ARK2/Frontend|Frontend]]><br />
|- sites/<br />
|- src/<br />
|- tests/<br />
|- var/<br />
|- cache/<br />
|- logs/<br />
|- vendor/<br />
<br />
* The 'bin' folder holds any executable binaries, primarily the admin console script<br />
* The 'build' folder holds tools and source files for building [[ARK2/Frontend|frontends]]<br />
* The 'sites' folder holds the site specific files in a way that supports multi-site installs<br />
* The 'src' folder holds the ARK source code<br />
* The 'tests' folder holds the ARK auto-tests<br />
* The 'var' folder holds transitory site files, such as caches and logs<br />
* The 'vendor' holds the component library code managed by Composer<br />
<br />
Tarball packaging for release will not include the 'build' or 'test' folders, nor the contents of the 'vendor' folder.<br />
<br />
The ARK2 'src' folder is organised as follows:<br />
<br />
/<br />
|- src/<br />
|- ARK/<br />
|- server/<br />
|- php/<br />
|- schema/<br />
|- database/<br />
|- json/<br />
|- web/<br />
|- <see [[ARK2/Frontend|Frontend]]><br />
|- <project>/<br />
<br />
* The 'src/ARK' folder holds the ARK Project specific code, this code is not intended for custom site code and will be over-written by any ARK update<br />
* The 'src/ARK/server' folder holds the ARK Server code, i.e. the backend code, API, database, etc<br />
* The 'src/ARK/web' folder holds the ARK Web code, i.e. the frontend code. See [[ARK2/Frontend|Frontend]] for more details.<br />
* The 'src/<project>' folder would hold custom frontend code where another project wishes to run their own multi-site install with a single custom frontend, see [[ARK2/Frontend|Frontend]] for more details<br />
<br />
The ARK2 'sites' folder is organised as follows:<br />
<br />
/<br />
|- sites/<br />
|- <site>/<br />
|- config/<br />
|- database.json<br />
|- site.json<br />
|- files<br />
|- <see [[ARK2/Files|Files]]><br />
|- src/<br />
|- <see [[ARK2/Frontend|Frontend]]><br />
|- schema/<br />
|- web/<br />
|- .htaccess<br />
|- index.php<br />
|- assets/<br />
|- <see [[ARK2/Frontend|Frontend]]><br />
<br />
* The 'sites/<site>' folder holds all the custom files required for a site. This allows for multi-site installs, as well as a simple site backup/transfer strategy. The <site> name is recommended to be the reverse domain name for the site if unique (e.g. com.example.www) or the subfolder if using a shared domain (e.g. mysite).<br />
* The 'sites/<site>/config' folder holds all the configuration files for the site.<br />
* The 'sites/<site>/files' folder holds all the data files for the site, such as photos and documents. See [[ARK2/Files|Files]] for more details.<br />
* The 'sites/<site>/src' folder holds any custom frontend code, see [[ARK2/Frontend|Frontend]] for more details.<br />
* The 'sites/<site>/schema' folder holds the site specific JSON Schema files.<br />
* The 'sites/<site>/web' folder is the webroot for the site, isolating only the front-controller and assets in this folder improves security.<br />
* The 'sites/<site>/web/assets' folder holds the assets for the frontend(s), usually as a symlink back to the 'src/ARK/web/<frontend>/assets' folder to allow for easy updates. See [[ARK2/Frontend|Frontend]] for more details.<br />
<br />
== Environment ==<br />
<br />
To develop ARK requires the following tools to be installed:<br />
* [https://git-scm.com/ Git]<br />
* [https://getcomposer.org/ Composer]<br />
* [https://nodejs.org/en/ Node.js] (for frontend/theme development only)<br />
<br />
You will also require PHP, a web server (Apache/PHP), a database server (MySQL/PostreSQL/SQLite), a web browser (Chrome/Firefox), and a an API client (Postman).<br />
<br />
The following tools are recommend to adhere to ARK code quality standards:<br />
* php-cs-fixer<br />
* PHP CodeSniffer<br />
* PHP MD<br />
<br />
=== macOS ===<br />
<br />
On macOS, while you can install the requirements via standalone packages, we recommend using HomeBrew as it makes installing and updating all the tools used easier.<br />
<br />
* Install XCode<br />
* Install the Command Line Tools (includes Git)<br />
* Download and install [http://brew.sh/ HomeBrew]<br />
* <pre>brew install node tidy-html5 homebrew/php/composer homebrew/completions/composer-completion homebrew/php/php-cs-fixer homebrew/php/php-code-sniffer homebrew/php/phpmd homebrew/php/sqlformat homebrew/php/phplint</pre><br />
* <pre>npm install -g bootlint csscomb csslint eslint js-beautify jsonlint prettydiff remark-lint sass-lint tidy-markdown</pre><br />
* If ypou need to do front end development you will also need <pre> gulp-sass gulp-sourcemaps</pre><br />
* (Optional) Modify your .profile to enable command-line auto-complete<br />
export HOMEBREW_GITHUB_API_TOKEN="token"<br />
source /Library/Developer/CommandLineTools/usr/share/git-core/git-completion.bash<br />
source /Library/Developer/CommandLineTools/usr/share/git-core/git-prompt.sh<br />
GIT_PS1_SHOWDIRTYSTATE=true<br />
PS1='\u@\h \W$(__git_ps1 " (%s)") \$ '<br />
export PATH=/Users/odysseus/.bin:$PATH<br />
<br />
NOTE add Homebrew to path!!!<br />
<br />
The easiest way to run a webserver and database server is using [https://www.mamp.info/en/ MAMP]. Simply install MAMP and create a soft-link from the MAMP webroot to the location of your ARK git repository. The alternative is to use HomeBrew to configure and run your own install. The HomeBrew method is recommended for developing with PostgreSQL (TODO).<br />
* <pre>brew install homebrew/php/php70 homebrew/php/php70-tidy homebrew/php/php70-xdebug</pre><br />
<br />
<br />
<br />
For QGIS Python:<br />
* 'brew install python qt pyqt'<br />
* 'pip install pep8 autopep8 jedi pb_tool '<br />
<br />
TODO<br />
* xdebug setup<br />
* apache / php /mysql / mamp<br />
* linter/fixer rulesets<br />
<br />
=== Atom Editor ===<br />
<br />
While text editors and IDEs are a deeply personal choice, we recommend using [https://atom.io/ Atom] as it is a cross-platform Open Source editor with powerful plugins to support the tools used by ARK. Using Atom ensures you have an environment consistent with the core ARK developers and ARK development standards.<br />
<br />
After installing Atom, check the commandline tools have been installed.<br />
<br />
Recommended Atom Packages:<br />
* 'apm install atom-autocomplete-php atom-beautify atom-bootstrap3 atom-symfony2 autocomplete-sass doctrine linter-bootlint linter-csslint linter-eslint linter-jsonlint linter-markdown linter-php linter-phpcs linter-phpmd linter-sass-lint linter-tidy linter-twig php-composer-completion php-cs-fixer php-debug php-twig symfony-snippets'<br />
<br />
Useful Atom Packages:<br />
* sublime-style-column-selection<br />
* autoclose-html<br />
* browser-plus<br />
* docblockr<br />
* git-plus<br />
* highlight-line<br />
* highlight-selected<br />
* minimap<br />
* open-recent<br />
* pigments<br />
* platformio-ide-terminal<br />
* project-manager<br />
* sort-lines<br />
* sync-settings<br />
<br />
== Install ==<br />
<br />
composer install</div>Mikehttps://ark.lparchaeology.com/wiki/index.php?title=Developer_Manual&diff=3726Developer Manual2016-10-21T16:40:23Z<p>Mike: </p>
<hr />
<div>The developer manual is aimed at people who will be contributing to the project and to a certain extent to those who intend to hack ARK to use it for their own ends. As a general rule, if you find yourself doing extensive hacking, please let us know as we would be happy to consider adding new code to the project.<br />
<br />
If you don't know why you are reading this documentation, you are probably looking in the wrong part of the manual. For information on installation and administration see the [[Administrator Manual| administrator]] section of the manual.<br />
<br />
Work to create developer documentation including migration of old documentation (Ticket #122) is ongoing (as of 21st Oct 2016) but in the meantime please contact a member of the [http://ark.lparchaeology.com/contact ARK Development Team] if you can't find what you are looking for within these pages.<br />
<br />
===Background Reading===<br />
<br />
====[[ARK model]]====<br />
An incomplete guide to the overall structure of ARK, its file structures etc.<br />
<br />
====[[ARK Data Storage]]====<br />
An overview of the ARK data storage model<br />
<br />
====[[Filters]]====<br />
An overview of filters and how they work (theoretical)<br />
<br />
===Standards and Guidance===<br />
<br />
Coding standards are simple and follow the PEAR system rigidly without exception. See the page blow for further detail. Some further guidance is given below to try to help you build new code in a way that fits into the overall ARK style. This material is incomplete at this stage.<br />
<br />
====[[Coding Standards]]====<br />
An overview of the coding standards with a link to PEAR.<br />
<br />
====[[Adding a New Page]]====<br />
How to add a new page to the system<br />
<br />
====[[Data Entry Pages]]====<br />
How the data entry pages fit together. Possibly needs to be re-written as something more generic<br />
<br />
====[[Page Header Block]]====<br />
example code for a page header block. Possibly needs to be made into a more generic code snippets page<br />
<br />
====[[Developing Subforms]]====<br />
A brief guide to developing new subforms<br />
<br />
====[[Accessing User Information]]====<br />
How to get access to the logged in user's information which is held in the SESSION<br />
<br />
====[[ARK log]]====<br />
An overview of the log system with detailed notes (note that this system is currently not fully working)<br />
<br />
====[[Markup and Aliases]]====<br />
An overview and guide<br />
<br />
====[[Error and Message Handling]]====<br />
An overview and guide<br />
<br />
====[[Configuration Settings for Developers]]====<br />
An overview for developers of the config system<br />
<br />
===Developer Documentation===<br />
<br />
The code is documented in detail within the header blocks for each file and the documentation blocks for each function. This doucmentation is in the process of being categorised and grouped for easier reading, but can be consulted at:<br />
<br />
====[[http://ark.lparchaeology.com/ark_phpdoc/li_ark.html PhpDocumenter Output]]====<br />
Detailed reference material for every function and script in the system - NB this is for ARK v0.7, and has not yet been updated for the latest release(s).<br />
<br />
====[[http://www.phpdoc.org/ PhpDocumenter]]====<br />
Some info on the software used to create the documentation from the header blocks<br />
<br />
[[Category:Developer]]<br />
<br />
===API Documentation===<br />
<br />
As of v1.1 it is possible to access ARK using an API via a URL - which currently responds with a JSON object.<br />
<br />
The following methods can be used:<br />
<br />
[[describeARK]] - this provides some simple metadata about the ARK instance<br />
<br />
[[describeItems]] - this provides a list of all the types of modules available<br />
<br />
[[describeFrags]] - this provides a list of all the different types of data available within the ARK instance (it lists the different 'types' of dataclass available<br />
<br />
[[describeFilters]] - this provides a list of all available filters<br />
<br />
[[describeSubforms]] - this provides a list of available subforms (for transclusion)<br />
<br />
[[describeFields]] - this provides a list of available fields<br />
<br />
[[getItems]] - this returns an array of items of a specified module<br />
<br />
[[getFrags]] - this returns an array of data frags attached to a specified item<br />
<br />
[[getFilter]] - this returns the result of a saved filter or a free-text search<br />
<br />
[[getFields]] - this returns an array of the fields attached to an item with their values<br />
<br />
[[getConcepts]] - this is a wrapper page which retrieves the definition of a particular concept<br />
<br />
[[transcludeFilter]] - this returns a fully rendered HTML representation of a filter result<br />
<br />
[[transcludeSubform]] - this returns a fully rendered HTML representation of a subform<br />
<br />
[[putField]] - this is for putting values to an ARK. Use this for adding, editing or deleting fields<br />
<br />
[[XMLParser]] - this is mainly used internally by the spatial elements of ARK to retrieve and present GML data<br />
<br />
<br />
[[Category:Developer]]</div>Mikehttps://ark.lparchaeology.com/wiki/index.php?title=Importing_Text_Data&diff=3355Importing Text Data2015-12-09T10:29:17Z<p>Mike: /* Step 3. Determine ARK IDs */</p>
<hr />
<div>==ARK textfile importer user guide==<br />
Since v1.1.2 ARK ahs had a tool to import text based files. in either JSON or CSV <br />
Data represented in either json or csv format can be uploaded to the ARK database.The file can be filtered so that only certain records are uploaded. It can create new records or add to existing ones.<br />
<br />
===Step 1. Identify datafile===<br />
[[File:jsonhowto_fig1.png|thumb|400px|right|Figure 1: File Picker]]<br />
Specify the location of the file.The import page of ARK in 1.1.2 includes a function for importing data directly from a text file or a data stream available online. Currently the file must be available on a web server. (Fig. 1) Enter the location of the file, either a relative path from the ARK root eg data/uploads/your_data.csv or a remote web address eg http://www.example.com/ARK/data?format=json.<br />
Click Submit.<br />
<br />
Depend on the size of the file it may take some time to load into your browser. Once it is loaded it will be available to manipulate very quickly.A spinner will appear while this is happening.<br />
<br />
===Step 2. Determine 'Root' of the data structure.===<br />
<br />
The data you have will likely include information that relates to several ARK items. Within your data structure there will be a level that includes all of these item representations - in the same way as rows in a table. In a csv file where there is a single record per row the root will be 'root>items'. In a more complicated json file (for example one with file level metadata held in the root as well as the items) the records may be contained in an object within the root object. Json allows for extensible representation, so each 'row' in the json file may not include all the same fields. It may be necessary to use the filter tools to only import objects which have a certain criteria.<br />
<br />
The example shown here has many items in the items object in root. The first available column is used to identify each of the contained objects - in this case it is an ARK ID, but it could be anything and may not be unique if it is not unique in your structure. For example if field 1 in your table is a site code or context type.<br />
<br />
[[File:jsonhowto_fig2.png|thumb|400px|left|none|Figure 2: Objects in 'root>items']]<br />
<br />
In order to define this level you will need to navigate to it.The path at the top of the page describes where in the data structure you currently are.The large panel below the advanced options shows the identifiers held in the current level.At the root level of a CSV file the first column will be used as an identifier. Clicking on these will open the data attached to that identifier for viewing.(Fig. 2) It is possible to go up one level using the button at the end of the list of objects, or any part of the location at the top of the page can be used to navigate to that level.When the main window would contain more than 20 items it is truncated and only the first 20 are shown.<br />
<br />
Use the 'this is root' button in the navigation panel at the top of the page when the root is displayed in the main panel.<br />
<br />
===Step 3. Determine ARK IDs===<br />
<br />
[[File:jsonhowto_fig3.png|thumb|right|600px|Figure 3: Root and ARK ID specified]]<br />
<br />
Using the navigation method above find the path to the ARK id within your data. This will often be as simple as 'root>items>item1>Ark_id'.The object that you define the ARK ID in is not important. As long as the objects are structured consistently the ARK IDs will be retrieved automatically based on the root defined in the step above. so for instance the importer will import the ARK ID for METSUR_152 from root > items > METSUR_152 > ARK_id and then the ARK ID for METSUR_153 from root > items > METSUR_153 > ARK_id, and so on.<br />
<br />
<br />
If you do not have ARK IDs in your data refer to the [[#Advanced Options|advanced options]].<br />
Otherwise, click 'This is ARK ID' button below the identifier that contains the ARK ID.<br />
<br />
===Step 4. Test Import===<br />
<br />
[[File:jsonhowto_fig4.png|thumb|right|400px|Figure 4: Example of dry run]]<br />
<br />
Click 'Import this' below the identifier that you would like to import. If you have completed the steps above a new panel will appear below the viewer. Confirm that this looks how you expect – '''this is what will be imported into the database'''.<br />
<br />
The final panel allows you to define which records will be imported. A type must be specified. If you are importing a file it may be necessary to do a separate import of each type. This is because different types have different fields attached to them, the drop down menus respond to the ones above them, presenting only the options available. If they are not completed in order it may result in unexpected behaviour.<br />
<br />
===Step 5.===<br />
When you are happy that you are importing the correct data into the correct field, click 'Submit'.<br />
You will be shown the results of your import. Click 'Import Json' to repeat the process for any other fields you wish to import.<br />
<br />
==Advanced Options==<br />
;Ste_cd:<br />
:The site code that imported items will have, is one is not specified in the data – uses the ARK default if not set. No site code: This check box will remove site codes from the import itemvalue, for use with chains.<br />
;Regular Expression:<br />
:This is used to extract the itemvalue from the ARK ID column – by default it grabs the first run of numbers in the field.<br />
;Start at Arbitrary Number:<br />
:If you wish to create a sequence of numbers starting from a given point, (including 1) you will need to specify this here, and the number to start on. A unique 'ARK ID' must still be specified in the data, but it will not be used.</div>Mikehttps://ark.lparchaeology.com/wiki/index.php?title=Importing_Text_Data&diff=3354Importing Text Data2015-12-09T10:25:35Z<p>Mike: /* Step 2. Determine 'Root' of the data structure. */</p>
<hr />
<div>==ARK textfile importer user guide==<br />
Since v1.1.2 ARK ahs had a tool to import text based files. in either JSON or CSV <br />
Data represented in either json or csv format can be uploaded to the ARK database.The file can be filtered so that only certain records are uploaded. It can create new records or add to existing ones.<br />
<br />
===Step 1. Identify datafile===<br />
[[File:jsonhowto_fig1.png|thumb|400px|right|Figure 1: File Picker]]<br />
Specify the location of the file.The import page of ARK in 1.1.2 includes a function for importing data directly from a text file or a data stream available online. Currently the file must be available on a web server. (Fig. 1) Enter the location of the file, either a relative path from the ARK root eg data/uploads/your_data.csv or a remote web address eg http://www.example.com/ARK/data?format=json.<br />
Click Submit.<br />
<br />
Depend on the size of the file it may take some time to load into your browser. Once it is loaded it will be available to manipulate very quickly.A spinner will appear while this is happening.<br />
<br />
===Step 2. Determine 'Root' of the data structure.===<br />
<br />
The data you have will likely include information that relates to several ARK items. Within your data structure there will be a level that includes all of these item representations - in the same way as rows in a table. In a csv file where there is a single record per row the root will be 'root>items'. In a more complicated json file (for example one with file level metadata held in the root as well as the items) the records may be contained in an object within the root object. Json allows for extensible representation, so each 'row' in the json file may not include all the same fields. It may be necessary to use the filter tools to only import objects which have a certain criteria.<br />
<br />
The example shown here has many items in the items object in root. The first available column is used to identify each of the contained objects - in this case it is an ARK ID, but it could be anything and may not be unique if it is not unique in your structure. For example if field 1 in your table is a site code or context type.<br />
<br />
[[File:jsonhowto_fig2.png|thumb|400px|left|none|Figure 2: Objects in 'root>items']]<br />
<br />
In order to define this level you will need to navigate to it.The path at the top of the page describes where in the data structure you currently are.The large panel below the advanced options shows the identifiers held in the current level.At the root level of a CSV file the first column will be used as an identifier. Clicking on these will open the data attached to that identifier for viewing.(Fig. 2) It is possible to go up one level using the button at the end of the list of objects, or any part of the location at the top of the page can be used to navigate to that level.When the main window would contain more than 20 items it is truncated and only the first 20 are shown.<br />
<br />
Use the 'this is root' button in the navigation panel at the top of the page when the root is displayed in the main panel.<br />
<br />
===Step 3. Determine ARK IDs===<br />
<br />
[[File:jsonhowto_fig3.png|thumb|right|600px|Figure 3: Root and ARK ID specified]]<br />
<br />
Using the navigation method above find the path to the ARK id within your data. This will often be as simple as 'root>items>item1>Ark_id'.The object that you define the ARK ID in is not important. As long as the objects are structured consistently the ARK IDs will be retrieved automatically based on the root defined in the step above. so for instance the importer will import the ARK ID for METSUR_152 from root > items > METSUR_152 > ARK_id and then the ARK ID for METSUR_153 from root > items > METSUR_153 > ARK_id, and so on.<br />
<br />
<br />
If you do not have ARK IDs in your data refer to the advanced options.<br />
Otherwise, click 'This is ARK ID' button below the identifier that contains the ARK ID.<br />
<br />
===Step 4. Test Import===<br />
<br />
[[File:jsonhowto_fig4.png|thumb|right|400px|Figure 4: Example of dry run]]<br />
<br />
Click 'Import this' below the identifier that you would like to import. If you have completed the steps above a new panel will appear below the viewer. Confirm that this looks how you expect – '''this is what will be imported into the database'''.<br />
<br />
The final panel allows you to define which records will be imported. A type must be specified. If you are importing a file it may be necessary to do a separate import of each type. This is because different types have different fields attached to them, the drop down menus respond to the ones above them, presenting only the options available. If they are not completed in order it may result in unexpected behaviour.<br />
<br />
===Step 5.===<br />
When you are happy that you are importing the correct data into the correct field, click 'Submit'.<br />
You will be shown the results of your import. Click 'Import Json' to repeat the process for any other fields you wish to import.<br />
<br />
==Advanced Options==<br />
;Ste_cd:<br />
:The site code that imported items will have, is one is not specified in the data – uses the ARK default if not set. No site code: This check box will remove site codes from the import itemvalue, for use with chains.<br />
;Regular Expression:<br />
:This is used to extract the itemvalue from the ARK ID column – by default it grabs the first run of numbers in the field.<br />
;Start at Arbitrary Number:<br />
:If you wish to create a sequence of numbers starting from a given point, (including 1) you will need to specify this here, and the number to start on. A unique 'ARK ID' must still be specified in the data, but it will not be used.</div>Mikehttps://ark.lparchaeology.com/wiki/index.php?title=Importing_Data&diff=3353Importing Data2015-11-17T15:55:43Z<p>Mike: /* Structure Map - CMAP Structure */</p>
<hr />
<div>As of the v1.0 release of ARK there is a set of data import tools within ARK. These tools speed up and improve the process of creating a concordance map between the source data and an ARK database. Never the less, using the tools will require a working knowledge of ARK's data structures. If you are unable to import your data or would like training in the ARK import tools, please contact [http://www.lparchaeology.com/cms/about-lp/contact L - P : Archaeology] who provide both training and custom ARK installations.<br />
<br />
==Overview==<br />
<br />
The ARK import tools are designed to provide a means to import data from table based data. This can be either in the form of a series of tables from a relational database or from a spreadsheet. The tools map the fields in the data onto fields within the ARK data structure. Data can then be inspected before running an import.<br />
<br />
The tools provide some join functionality and are intended to reduce the need for pre-processing of data where possible. Inevitably, there will be some need to pre-process data for import.<br />
<br />
===Concordance Maps===<br />
<br />
Each instance of ARK can contain multiple concordance maps. The concordance map holds the information (mappings) that explains to the import tools how a particular element of the data should be reworked for import to the ARK structure. The concordance map maps both 'structure' and 'data'.<br />
<br />
====Structure Mapping====<br />
<br />
This is the key part of each concordance map, it contains the mappings for each field in the source data that will be imported into the ARK database. Due to the way ARK data is structured, it is often the case that many source data fields are not needed for import.<br />
<br />
Once mapped, a field can be imported and re-imported using the same mapping. Future version of ARK may support this as a method for dynamic updates of the data over the web.<br />
<br />
To use a simple example, a field in the source data such as 'Description' which is a column in a source database, will be mapped to an ARK txttype ready for import.<br />
<br />
====Data====<br />
<br />
Data mapping maps certain terms or values in a source database to certain terms or values in the ARK database. For example, this permits a term such as 'castle' in a source database to be mapped to a term such as 'defensive structure' in a target database.<br />
<br />
When importing from controlled lists, ARK makes new data mappings on the fly as it finds each new term within the list.<br />
<br />
By manually creating a data mapping, it is possible to alter the way this data is imported without the need for pre-processing. This is especially important in the case of dynamically re-importing data.<br />
<br />
==How to import data==<br />
<br />
This gives an overview of how to import data into ARK using the built in import tools.<br />
<br />
===Source Data===<br />
<br />
The first task is to take a close look at the data you are about to import. Problems in the data will not be magically corrected by the import tools. If you put junk in you will get junk out.<br />
<br />
At this stage the tools will only read data in from a source database on the same MySQL server. There is no reason you couldn't adapt the tools to easily work with another database such as Postgres, but at this stage, the tools will not read in data external to the server. The first task is therefore to import your data to the server. You can either create tables on the target database or set up a separate 'source database'.<br />
<br />
The easiest way to get data onto the server is probably to save each table as a comma separated values file and import this to the server using phpMyAdmin.<br />
<br />
Be sure to prefix source database tables with "import_" this will indicate to the import tools that you wish to map data in this table. Look up tables and other supplementary tables which will not be directly imported do not need this prefix.<br />
<br />
As a rule, you should name your columns with sensible names, avoid fancy characters and so on. Try to make the names unique and memorable.<br />
<br />
It is often simplest to create the table on the database first and then simply import the data into this table (and its defined columns), although this is up to you.<br />
<br />
Once you have imported your source data to the server, it is time to start mapping your databases.<br />
<br />
*'''UID Column''' - Any table that you prefix 'import_' and intend to import data from MUST have a column which contains a unique ID for each row of the table. This is used to loop over the data by the tools, it is not imported in anyway, but it is essential. This can in fact be a column containing your 'key' data, although in certain circumstances it is desirable to create (manually) and new UID column on the source table specifically for the purpose of importing the data. See below for further information.<br />
<br />
===Concordance Map - CMAP===<br />
<br />
If you have not done so already, you will need to create a new Concordance Map. To do this, use the tool available on the left panel fo the import tools home page.<br />
<br />
Fill in the required fields carefully (check spellings!) and save the new map to the database.<br />
<br />
*'''Nickname''' - A mnemonic, this can be anything you like that will help you remember the CMAP. Do NOT use spaces or funny characters here.<br />
*'''Description''' - A text field that you can use to describe this CMAP for future reference. This will accept UTF-8 characters etc.<br />
*'''Source DB''' - The exact name of the source database on this server. (May be the same as the ARK db)<br />
*'''Target Site Code''' - This is a default site code for the import. More complex options for this can be set in the structure map (which override this setting).<br />
<br />
You can edit this map afterwards using the built in edit CMAP tool.<br />
<br />
===Structure Map - CMAP Structure===<br />
<br />
Once you have set up a CMAP, you can begin mapping fields in the source DB. The key thing here is to treat each and every field (column) in the source data s an independent entity. The import tools look only at a single column, they do not try to recreate your relational database beyond a few specific functions.<br />
<br />
The recommended method is to look at each column in turn to decide if it will be imported into ARK or not. This is not as simple as it sounds so take time to analyse each column.<br />
<br />
In order to set up each field for import, use the tools provided. On the left panel of the import pages, select 'Edit Structure Map'. The tools provided list out all the 'import_' tables available on this CMAP in the right hand panel. In order to start working simple click the edit icon next to the table.<br />
<br />
This will then display each of the fields within this table that are available for import. Fields that are already mapped will be indicated as such. Each field may only be mapped once within a given CMAP, but could be mapped from multiple CMAPs if needed.<br />
<br />
In order to create a new mapping click the edit icon for the field. The tool will ask a couple of simple questions about the field before presenting a form.<br />
<br />
*'''Class''' - The first question asks what type of import the tools should run on this data. These classes mirror (but do NOT match) the ARK dataclasses.<br />
<br />
*'''Raw Item Val''' - The next question aims to establish the location of the raw item val. This is a key concept in the import tools (see notes). This question aims to establish whether a join (to a secondary table) will be necessary to arrive at this information. in the simplest case, this information is in a column on the import table itself. The item values to import must be in the sql.<br />
<br />
*'''Site Code''' - The system provides three options for the site code to be used in the import of each field. The simplest option is just to use the fixed site code specified in the CMAP. The second option is to use a code specified in a column of this table which will permit data from multiple sites to be imported from this field. Thirdly there is an option to get a site code from a joined table.<br />
<br />
In addition, the value 'notset' is available for most import types:<br />
<br />
*'''notset''' - This provides a way to specify records that should not be imported to the database. For example, you might use a keyword BLANK or SKIP in your source data as a way to prevent certain rows from being imported. Be aware that you should carefully check over the data to be imported in the extraction test.<br />
<br />
===Available Classes===<br />
<br />
====Keys====<br />
<br />
This class is used to import the keys into the ARK database. This tool will create the itemvalues for this module. Therefore this information needs only to be imported once. Effectively this import will 'register' all of the new records to the ARK database. In ARK there are modules that use modtypes and those that don't. The import tools can import keys for modules with or without keys.<br />
<br />
*'''Modkey''' - This imports keys to a module using modtypes. It does NOT import the modtype data. It will leave this column blank in the table. The modtype information MUST be imported to avoid nasty errors.<br />
<br />
*'''Key''' - This imports keys for modules not using modtypes.<br />
<br />
The data in THIS field/column will be turned into the ARK key by appending the site code to the front of it. You need to ensure that this data would be suitable to be used as an ARK key. It must not contain underscores for example. Duplicated numbers int his column will also cause the import to crash.<br />
<br />
The form will only ask for three things:<br />
<br />
*'''UID Column''' - see notes<br />
*'''Itemkey''' - The itemkey for which these values will become ARK itemvalues (defines the module)<br />
*'''not set''' - see notes<br />
<br />
====Boolean TRUE|FALSE|unset (attribute A)====<br />
<br />
This kind of true/false data is imported to ARK as attributes. In this case it is presupposed that every row will be imported.<br />
<br />
*'''true''' - Use this to set up which values will be turned into a logical positive in ARK<br />
*'''false''' - Use this to set up which values will be turned into a logical negative in ARK<br />
<br />
Before running the import, you should make sure that the attribute itself exists.<br />
<br />
Note that in ARK booleans are represented as 0 and 1 but that this may be aliased to whatever you like.<br />
<br />
====Controlled Lists (attribute B)====<br />
<br />
Data from controlled lists (aka 'look up tables', aka 'drop down menus') will be imported into ARK as class "attribute". For import purposes (only) there are two types of attribute. Type B attributes are of the kind where a series of terms in a controlled list are in play, these may or may not be applied to the item. It is NOT presupposed that a value must be set for every item. Multiple entries per item are also permitted (each one must be on its own unique row int he source data).<br />
<br />
Before beginning the import, you must establish the attributetype of the attribute in question, if not present, this must be added before attempting to create the mapping.<br />
<br />
Any attributes themselves may be manually set up before import or will be automatically added (and aliased) by the function if they are not already present int he controlled list for the specified attribute type.<br />
<br />
Note: All boolean values for any attributes added will be forced to TRUE (1) in the import.<br />
<br />
====Numbers, Text, Dates====<br />
<br />
These three types of data are simply imported as is. The previous comments on raw itemvals and 'notset' apply<br />
<br />
====Spans====<br />
<br />
Note that you must have manually set up the span type.<br />
<br />
Spans rely on having raw data in both the beginning field and end field. This data must be pre-processed as it will be entered into ark AS IS.<br />
<br />
Only add a mapping for the 'beginning' of spans. In this mapping the column containing the 'end' data is specified as 'end_source_col'. Do not map 'ends' as this will duplicate entry<br />
<br />
====Action====<br />
<br />
At present there is no user form for adding actions to the CMAP<br />
<br />
====Links to other modules - XMI====<br />
<br />
Specify the XMI itemkey and xmi itemval column for the XMI links.<br />
<br />
===Extraction Tests and Execution===<br />
<br />
Once you have registered a mapping into the database, this can be tested. The test will do a dry run on the import and show you a table of results that show the exact values to be added to the database for this extraction from the source db. It is vital that you very closely check this data to ensure that you have got your set up correct.<br />
<br />
In addition, check for duplicates or otherwise mangled data.<br />
<br />
Undoing an incorrectly specified import will require use of the phpMyAdmin tool and will require knowledge of SQL.<br />
<br />
Bear in mind that in the case of Attributes of type B, the preview will most likely show a series of SQL statements indicating the additions to the look up table (and Aliases) that would be run.<br />
<br />
===Importing Chains===<br />
<br />
Chains can be imported by using the link in the structure defining frame: <br />
<br />
:If attributes are to be chained to this number then click <u>[here]</u>.<br />
<br />
Clicking that link will save the structure and flag it so that when it is imported the ids of the new datafrags will be inserted into a column in the import_ table named ''''%col%'''_itemval', where '''%col%''' is the name of the column to chain.<br />
<br />
[[category:administrator]]</div>Mikehttps://ark.lparchaeology.com/wiki/index.php?title=Importing_Text_Data&diff=3352Importing Text Data2015-11-17T08:57:07Z<p>Mike: /* Advanced Options */</p>
<hr />
<div>==ARK textfile importer user guide==<br />
Since v1.1.2 ARK ahs had a tool to import text based files. in either JSON or CSV <br />
Data represented in either json or csv format can be uploaded to the ARK database.The file can be filtered so that only certain records are uploaded. It can create new records or add to existing ones.<br />
<br />
===Step 1. Identify datafile===<br />
[[File:jsonhowto_fig1.png|thumb|400px|right|Figure 1: File Picker]]<br />
Specify the location of the file.The import page of ARK in 1.1.2 includes a function for importing data directly from a text file or a data stream available online. Currently the file must be available on a web server. (Fig. 1) Enter the location of the file, either a relative path from the ARK root eg data/uploads/your_data.csv or a remote web address eg http://www.example.com/ARK/data?format=json.<br />
Click Submit.<br />
<br />
Depend on the size of the file it may take some time to load into your browser. Once it is loaded it will be available to manipulate very quickly.A spinner will appear while this is happening.<br />
<br />
===Step 2. Determine 'Root' of the data structure.===<br />
<br />
The data you have will likely include information that relates to several ARK items. Within your data structure there will be a level that includes all of these item representations - in the same way as rows in a table. In a csv file where there is a single record per row the root will be 'root>items'. In a more complicated json file (for example one with file level metadata held in the root as well as the items) the records may be contained in an object within the root object. Json allows for extensible representation, so each 'row' in the json file may not include all the same fields. It may be necessary to use the filter tools to only import objects which have a certain criteria.<br />
<br />
The example shown here has many items in the items object in root. The first available row is used to identify each of the contained objects - in this case it is an ARK ID, but it could be anything and may not be unique if it is not unique in your structure. For example if field 1 in your table is a site code or context type.<br />
<br />
[[File:jsonhowto_fig2.png|thumb|400px|left|none|Figure 2: Objects in 'root>items']]<br />
<br />
In order to define this level you will need to navigate to it.The path at the top of the page describes where in the data structure you currently are.The large panel below the advanced options shows the identifiers held in the current level.At the root level of a CSV file the first column will be used as an identifier. Clicking on these will open the data attached to that identifier for viewing.(Fig. 2) It is possible to go up one level using the button at the end of the list of objects, or any part of the location at the top of the page can be used to navigate to that level.When the main window would contain more than 20 items it is truncated and only the first 20 are shown.<br />
<br />
Use the 'this is root' button in the navigation panel at the top of the page when the root is displayed in the main panel.<br />
<br />
===Step 3. Determine ARK IDs===<br />
<br />
[[File:jsonhowto_fig3.png|thumb|right|600px|Figure 3: Root and ARK ID specified]]<br />
<br />
Using the navigation method above find the path to the ARK id within your data. This will often be as simple as 'root>items>item1>Ark_id'.The object that you define the ARK ID in is not important. As long as the objects are structured consistently the ARK IDs will be retrieved automatically based on the root defined in the step above. so for instance the importer will import the ARK ID for METSUR_152 from root > items > METSUR_152 > ARK_id and then the ARK ID for METSUR_153 from root > items > METSUR_153 > ARK_id, and so on.<br />
<br />
<br />
If you do not have ARK IDs in your data refer to the advanced options.<br />
Otherwise, click 'This is ARK ID' button below the identifier that contains the ARK ID.<br />
<br />
===Step 4. Test Import===<br />
<br />
[[File:jsonhowto_fig4.png|thumb|right|400px|Figure 4: Example of dry run]]<br />
<br />
Click 'Import this' below the identifier that you would like to import. If you have completed the steps above a new panel will appear below the viewer. Confirm that this looks how you expect – '''this is what will be imported into the database'''.<br />
<br />
The final panel allows you to define which records will be imported. A type must be specified. If you are importing a file it may be necessary to do a separate import of each type. This is because different types have different fields attached to them, the drop down menus respond to the ones above them, presenting only the options available. If they are not completed in order it may result in unexpected behaviour.<br />
<br />
===Step 5.===<br />
When you are happy that you are importing the correct data into the correct field, click 'Submit'.<br />
You will be shown the results of your import. Click 'Import Json' to repeat the process for any other fields you wish to import.<br />
<br />
==Advanced Options==<br />
;Ste_cd:<br />
:The site code that imported items will have, is one is not specified in the data – uses the ARK default if not set. No site code: This check box will remove site codes from the import itemvalue, for use with chains.<br />
;Regular Expression:<br />
:This is used to extract the itemvalue from the ARK ID column – by default it grabs the first run of numbers in the field.<br />
;Start at Arbitrary Number:<br />
:If you wish to create a sequence of numbers starting from a given point, (including 1) you will need to specify this here, and the number to start on. A unique 'ARK ID' must still be specified in the data, but it will not be used.</div>Mikehttps://ark.lparchaeology.com/wiki/index.php?title=Importing_Text_Data&diff=3351Importing Text Data2015-11-17T08:50:48Z<p>Mike: </p>
<hr />
<div>==ARK textfile importer user guide==<br />
Since v1.1.2 ARK ahs had a tool to import text based files. in either JSON or CSV <br />
Data represented in either json or csv format can be uploaded to the ARK database.The file can be filtered so that only certain records are uploaded. It can create new records or add to existing ones.<br />
<br />
===Step 1. Identify datafile===<br />
[[File:jsonhowto_fig1.png|thumb|400px|right|Figure 1: File Picker]]<br />
Specify the location of the file.The import page of ARK in 1.1.2 includes a function for importing data directly from a text file or a data stream available online. Currently the file must be available on a web server. (Fig. 1) Enter the location of the file, either a relative path from the ARK root eg data/uploads/your_data.csv or a remote web address eg http://www.example.com/ARK/data?format=json.<br />
Click Submit.<br />
<br />
Depend on the size of the file it may take some time to load into your browser. Once it is loaded it will be available to manipulate very quickly.A spinner will appear while this is happening.<br />
<br />
===Step 2. Determine 'Root' of the data structure.===<br />
<br />
The data you have will likely include information that relates to several ARK items. Within your data structure there will be a level that includes all of these item representations - in the same way as rows in a table. In a csv file where there is a single record per row the root will be 'root>items'. In a more complicated json file (for example one with file level metadata held in the root as well as the items) the records may be contained in an object within the root object. Json allows for extensible representation, so each 'row' in the json file may not include all the same fields. It may be necessary to use the filter tools to only import objects which have a certain criteria.<br />
<br />
The example shown here has many items in the items object in root. The first available row is used to identify each of the contained objects - in this case it is an ARK ID, but it could be anything and may not be unique if it is not unique in your structure. For example if field 1 in your table is a site code or context type.<br />
<br />
[[File:jsonhowto_fig2.png|thumb|400px|left|none|Figure 2: Objects in 'root>items']]<br />
<br />
In order to define this level you will need to navigate to it.The path at the top of the page describes where in the data structure you currently are.The large panel below the advanced options shows the identifiers held in the current level.At the root level of a CSV file the first column will be used as an identifier. Clicking on these will open the data attached to that identifier for viewing.(Fig. 2) It is possible to go up one level using the button at the end of the list of objects, or any part of the location at the top of the page can be used to navigate to that level.When the main window would contain more than 20 items it is truncated and only the first 20 are shown.<br />
<br />
Use the 'this is root' button in the navigation panel at the top of the page when the root is displayed in the main panel.<br />
<br />
===Step 3. Determine ARK IDs===<br />
<br />
[[File:jsonhowto_fig3.png|thumb|right|600px|Figure 3: Root and ARK ID specified]]<br />
<br />
Using the navigation method above find the path to the ARK id within your data. This will often be as simple as 'root>items>item1>Ark_id'.The object that you define the ARK ID in is not important. As long as the objects are structured consistently the ARK IDs will be retrieved automatically based on the root defined in the step above. so for instance the importer will import the ARK ID for METSUR_152 from root > items > METSUR_152 > ARK_id and then the ARK ID for METSUR_153 from root > items > METSUR_153 > ARK_id, and so on.<br />
<br />
<br />
If you do not have ARK IDs in your data refer to the advanced options.<br />
Otherwise, click 'This is ARK ID' button below the identifier that contains the ARK ID.<br />
<br />
===Step 4. Test Import===<br />
<br />
[[File:jsonhowto_fig4.png|thumb|right|400px|Figure 4: Example of dry run]]<br />
<br />
Click 'Import this' below the identifier that you would like to import. If you have completed the steps above a new panel will appear below the viewer. Confirm that this looks how you expect – '''this is what will be imported into the database'''.<br />
<br />
The final panel allows you to define which records will be imported. A type must be specified. If you are importing a file it may be necessary to do a separate import of each type. This is because different types have different fields attached to them, the drop down menus respond to the ones above them, presenting only the options available. If they are not completed in order it may result in unexpected behaviour.<br />
<br />
===Step 5.===<br />
When you are happy that you are importing the correct data into the correct field, click 'Submit'.<br />
You will be shown the results of your import. Click 'Import Json' to repeat the process for any other fields you wish to import.<br />
<br />
==Advanced Options==<br />
#Ste_cd:The site code that imported items will have, is one is not specified in the data – uses the ARK default if not set. No site code: This check box will remove site codes from the import itemvalue, for use with chains.<br />
#Regular Expression: This is used to extract the itemvalue from the ARK ID column – by default it grabs the first run of numbers in the field.<br />
#Start at Arbitrary Number: If you wish to create a sequence of numbers starting from a given point, (including 1) you will need to specify this here, and the number to start on. A unique 'ARK ID' must still be specified in the data, but it will not be used.</div>Mikehttps://ark.lparchaeology.com/wiki/index.php?title=Administrator_Manual&diff=3350Administrator Manual2015-11-16T08:45:29Z<p>Mike: /* Installation and Setup */</p>
<hr />
<div>__TOC__<br />
<br />
The administrator manual is aimed at people who are downloading and installing ARK. If you are struggling with installation or administration, you should consider contacting the ARK team via the [http://groups.google.com/group/arkusers Google Group].<br />
<br />
===Key Concepts===<br />
<br />
====[[Navigation]]====<br />
<br />
Information about the navigation tools used within ARK.<br />
<br />
====[[Views]]====<br />
<br />
The different 'Views' of ARK.<br />
<br />
====[[Key Administration Concepts]]====<br />
<br />
An overview of the key administration concepts.<br />
<br />
===Installation and Setup===<br />
<br />
====[[Basic Installation]]====<br />
<br />
A description of the ARK installation process.<br />
<br />
====[[Database Tables]]====<br />
<br />
A list of the standard ARK tables, with a basic explanation for each.<br />
<br />
===ARK Customisation===<br />
<br />
This chapter will help the new ARK administrator who is comfortable with editing PHP configuration files through the setup of an ARK project. For a general overview of settings please see [[Key Administration Concepts#Settings Files|Settings Files]].<br />
<br />
====[[env_settings.php]]====<br />
<br />
A description of the env_settings.php file for three different operating systems.<br />
<br />
====[[field_settings.php]]====<br />
<br />
Instructions on how to configure the field_settings.php file.<br />
<br />
====[[mod_settings.php]]====<br />
<br />
Instructions on how to configure module specific settings files.<br />
<br />
====[[settings.php]]====<br />
<br />
The general settings for ARK.<br />
<br />
====[[page_settings.php]]====<br />
<br />
The settings for page based displays and navigation. <br />
<br />
====[[vd_settings.php]]====<br />
<br />
A description of information found within the vd_settings.php file.<br />
<br />
===Subforms===<br />
<br />
====[[Subforms Overview]]====<br />
<br />
A general overview of ARK Subforms and their configuration.<br />
<br />
====[[Subform Requirements]]====<br />
<br />
The required fields for configuration of subforms.<br />
<br />
====[[Subform Options]]====<br />
<br />
Some optional subform fields generic to all modules and subform types.<br />
<br />
====[[Conditional Subforms]]====<br />
<br />
Conditional subforms is a more complex subform option that allows specific subforms to only be displayed if a set of pre-configured conditions are met<br />
<br />
====[[Subform Reference]]====<br />
<br />
A list of all the current subforms with example configurations.<br />
<br />
===Mapping===<br />
<br />
====[[Mapping Overview]]====<br />
Some general information about mapping in ARK.<br />
<br />
====[[Configuring the Mapfile]]====<br />
Instructions on how to configure the mapfile.<br />
<br />
====[[OpenStreetMap and ARK]]====<br />
Instructions on how to use OpenStreetmap as your mapping BaseLayer<br />
<br />
====[[Google Maps and ARK]]====<br />
Instructions on how to use Google Maps as your mapping BaseLayer<br />
<br />
===Administration===<br />
<br />
====[[User Administration]]====<br />
<br />
Some tools for administering user accounts.<br />
<br />
====[[Address Book Administration]]====<br />
<br />
How to update the Address Book module.<br />
<br />
====[[Alias Administration]]====<br />
<br />
How to add aliases to the database.<br />
<br />
====[[Markup Administration]]====<br />
<br />
How to add markup to the database.<br />
<br />
====[[Exporting Data]]====<br />
<br />
Some basic tips for exporting data.<br />
<br />
====[[Importing Data]]====<br />
<br />
Information about the ongoing development of import tools.<br />
<br />
====[[The ARK Help System]]====<br />
<br />
How to use and administer the Help system.<br />
<br />
====[[Setting up the Wordpress Plugin]]====<br />
<br />
This is how to setup and use the ARK Wordpress plugin<br />
<br />
===Configuring Clean URLs===<br />
<br />
ARK can be configured to use clean URLs, that is instead of visiting a page using a URL like: http://www.myark.com/ark/micro_view.php?item_key=cxt_cd&itemvalue=ABC12_1234 you can visit: http://www.myark.com/ark/view/cxt/ABC12_1234. This allows for persistent clean URLs and also aids in Search Engine Optimisation, etc. If you are still not sure you want to do this, [http://www.hochmanconsultants.com/articles/clean-urls.shtml here] is an article explaining it a bit further. <br />
<br />
If you still want to do it there are a number of steps to go through:<br />
<br />
1. [[Set up your webserver to enable mod_rewrite]]<br />
<br />
2. [[Set up your Rewrite Rules]]<br />
<br />
3. [[Check the setup and adjust your main pages]]<br />
<br />
===Custom Configuration===<br />
<br />
The following are examples of custom configurations and may be specific to previous installations of ARK. Only use these for reference if you are upgrading from development code prior to the v0.6 release. Please contact the [http://ark.lparchaeology.com/contact ARK Development Team] for more information on any of the following. <br />
<br />
====[[Transcluding]]====<br />
<br />
====[[Configuring file registration]]====<br />
<br />
====[[Configuring the Mapping]]====<br />
<br />
====[[Configuring RSS Feeds]]====<br />
<br />
[[category: Administrator]]</div>Mikehttps://ark.lparchaeology.com/wiki/index.php?title=Sf_chaintable&diff=3349Sf chaintable2015-10-21T14:48:57Z<p>Mike: /* Example Configuration */</p>
<hr />
<div>===Description===<br />
<br />
sf_chaintable can be used for displaying and editing data fragments that are chained to other data fragments. One 'root' field is linked to the item being viewed, which may have one or more fragments per item creating rows for the table and then one or more fragments can be chained to these to create additional columns. Examples of this include attaching numbers to find attributes - mirroring and supplements [[sf_assemblage]] for linking usage to periods of sites, or for levels tables.<br />
<br />
By default the chaintable hides any empty columns. As of version 1.2 the new records feature is incomplete and only some of the edit functions are supported - numbers, text and attributes.<br />
<br />
===Additional Fields===<br />
<br />
The sf_assemblage needs a number of different 'op' variables set in the sf_conf:<br />
*'''op_assemblage_type''' - the root field which is connected to the item<br />
*'''op_sum_field''' - if this is set then the rows will be aggregated where they are identical and the number in this field will be summed for the output. this field must be a number!<br />
*'''fields''' - the columns that will be added to the table<br />
<br />
===Example Configuration===<br />
<pre><br />
$conf_mcd_assemblage =<br />
array(<br />
'view_state' => 'max',<br />
'edit_state' => 'view',<br />
'sf_title' => 'assemblage', //appears in the titlebar of the subform (mk nname)<br />
'sf_html_id' => 'pottery_ass', //the form id tag (must be unique)<br />
'sf_nav_type' => 'nmedit',<br />
'script' => 'php/subforms/sf_chaintable.php',<br />
'op_assemblage_type' => $conf_field_sherd_class,// the field of data attached to the item<br />
'op_sum_field' => $conf_field_total,<br />
'default' => '0', //the default value for creating new records<br />
'op_label' => 'space',<br />
'op_input' => 'save',<br />
'fields' => array(<br />
$conf_field_sherd_class,<br />
$conf_field_sherd_form,<br />
$conf_field_sherd_type,<br />
$conf_field_finddates,<br />
$conf_field_sherdfunction,<br />
$conf_field_catxmiloc,<br />
$conf_field_total,<br />
),<br />
);<br />
</pre><br />
<br />
<br />
[[category: Administrator]]</div>Mikehttps://ark.lparchaeology.com/wiki/index.php?title=Chains&diff=3348Chains2015-10-13T15:18:49Z<p>Mike: </p>
<hr />
<div>Chains are used in the database when you want a field to be attached to another field instead of an item. An example is the interpretation subform which contains a piece of text, a date and an action. The text-type 'interpretation' will be attached to the item and the date and action will be attached to the piece of text. [[File:chain_example.jpg|right]]<br />
<br />
The tables in this image are a very simplified schematic of how the data will be stored in ARK. You can see that a chain uses a different table rather than an itemkey as the itemkey for a data fragment. This example shows data the could be displayed as<br />
<br />
{| style="border-style: solid; border-width: 2px; border-collapse:collapse"<br />
|+Pottery classification<br />
|-<br />
|Assemblage<br />
|Class<br />
|Count<br />
|-<br />
|1102<br />
|Black Gloss Ware<br />
|3<br />
|-<br />
|1103<br />
|Black Gloss Ware<br />
|1<br />
|-<br />
|1102<br />
|Archaic Fine Ware<br />
|2<br />
|-<br />
|}<br />
<br />
[[Category:Glossary]]</div>Mikehttps://ark.lparchaeology.com/wiki/index.php?title=File:Chain_example.jpg&diff=3347File:Chain example.jpg2015-10-13T14:31:18Z<p>Mike: An example of a chain in ARK</p>
<hr />
<div>An example of a chain in ARK</div>Mikehttps://ark.lparchaeology.com/wiki/index.php?title=Importing_Data&diff=3346Importing Data2015-10-13T14:24:15Z<p>Mike: /* Importing Chains */</p>
<hr />
<div>As of the v1.0 release of ARK there is a set of data import tools within ARK. These tools speed up and improve the process of creating a concordance map between the source data and an ARK database. Never the less, using the tools will require a working knowledge of ARK's data structures. If you are unable to import your data or would like training in the ARK import tools, please contact [http://www.lparchaeology.com/cms/about-lp/contact L - P : Archaeology] who provide both training and custom ARK installations.<br />
<br />
==Overview==<br />
<br />
The ARK import tools are designed to provide a means to import data from table based data. This can be either in the form of a series of tables from a relational database or from a spreadsheet. The tools map the fields in the data onto fields within the ARK data structure. Data can then be inspected before running an import.<br />
<br />
The tools provide some join functionality and are intended to reduce the need for pre-processing of data where possible. Inevitably, there will be some need to pre-process data for import.<br />
<br />
===Concordance Maps===<br />
<br />
Each instance of ARK can contain multiple concordance maps. The concordance map holds the information (mappings) that explains to the import tools how a particular element of the data should be reworked for import to the ARK structure. The concordance map maps both 'structure' and 'data'.<br />
<br />
====Structure Mapping====<br />
<br />
This is the key part of each concordance map, it contains the mappings for each field in the source data that will be imported into the ARK database. Due to the way ARK data is structured, it is often the case that many source data fields are not needed for import.<br />
<br />
Once mapped, a field can be imported and re-imported using the same mapping. Future version of ARK may support this as a method for dynamic updates of the data over the web.<br />
<br />
To use a simple example, a field in the source data such as 'Description' which is a column in a source database, will be mapped to an ARK txttype ready for import.<br />
<br />
====Data====<br />
<br />
Data mapping maps certain terms or values in a source database to certain terms or values in the ARK database. For example, this permits a term such as 'castle' in a source database to be mapped to a term such as 'defensive structure' in a target database.<br />
<br />
When importing from controlled lists, ARK makes new data mappings on the fly as it finds each new term within the list.<br />
<br />
By manually creating a data mapping, it is possible to alter the way this data is imported without the need for pre-processing. This is especially important in the case of dynamically re-importing data.<br />
<br />
==How to import data==<br />
<br />
This gives an overview of how to import data into ARK using the built in import tools.<br />
<br />
===Source Data===<br />
<br />
The first task is to take a close look at the data you are about to import. Problems in the data will not be magically corrected by the import tools. If you put junk in you will get junk out.<br />
<br />
At this stage the tools will only read data in from a source database on the same MySQL server. There is no reason you couldn't adapt the tools to easily work with another database such as Postgres, but at this stage, the tools will not read in data external to the server. The first task is therefore to import your data to the server. You can either create tables on the target database or set up a separate 'source database'.<br />
<br />
The easiest way to get data onto the server is probably to save each table as a comma separated values file and import this to the server using phpMyAdmin.<br />
<br />
Be sure to prefix source database tables with "import_" this will indicate to the import tools that you wish to map data in this table. Look up tables and other supplementary tables which will not be directly imported do not need this prefix.<br />
<br />
As a rule, you should name your columns with sensible names, avoid fancy characters and so on. Try to make the names unique and memorable.<br />
<br />
It is often simplest to create the table on the database first and then simply import the data into this table (and its defined columns), although this is up to you.<br />
<br />
Once you have imported your source data to the server, it is time to start mapping your databases.<br />
<br />
*'''UID Column''' - Any table that you prefix 'import_' and intend to import data from MUST have a column which contains a unique ID for each row of the table. This is used to loop over the data by the tools, it is not imported in anyway, but it is essential. This can in fact be a column containing your 'key' data, although in certain circumstances it is desirable to create (manually) and new UID column on the source table specifically for the purpose of importing the data. See below for further information.<br />
<br />
===Concordance Map - CMAP===<br />
<br />
If you have not done so already, you will need to create a new Concordance Map. To do this, use the tool available on the left panel fo the import tools home page.<br />
<br />
Fill in the required fields carefully (check spellings!) and save the new map to the database.<br />
<br />
*'''Nickname''' - A mnemonic, this can be anything you like that will help you remember the CMAP. Do NOT use spaces or funny characters here.<br />
*'''Description''' - A text field that you can use to describe this CMAP for future reference. This will accept UTF-8 characters etc.<br />
*'''Source DB''' - The exact name of the source database on this server. (May be the same as the ARK db)<br />
*'''Target Site Code''' - This is a default site code for the import. More complex options for this can be set in the structure map (which override this setting).<br />
<br />
You can edit this map afterwards using the built in edit CMAP tool.<br />
<br />
===Structure Map - CMAP Structure===<br />
<br />
Once you have set up a CMAP, you can begin mapping fields in the source DB. The key thing here is to treat each and every field (column) in the source data s an independent entity. The import tools look only at a single column, they do not try to recreate your relational database beyond a few specific functions.<br />
<br />
The recommended method is to look at each column in turn to decide if it will be imported into ARK or not. This is not as simple as it sounds so take time to analyse each column.<br />
<br />
In order to set up each field for import, use the tools provided. On the left panel of the import pages, select 'Edit Structure Map'. The tools provided list out all the 'import_' tables available on this CMAP in the right hand panel. In order to start working simple click the edit icon next to the table.<br />
<br />
This will then display each of the fields within this table that are available for import. Fields that are already mapped will be indicated as such. Each field may only be mapped once within a given CMAP, but could be mapped from multiple CMAPs if needed.<br />
<br />
In order to create a new mapping click the edit icon for the field. The tool will ask a couple of simple questions about the field before presenting a form.<br />
<br />
*'''Class''' - The first question asks what type of import the tools should run on this data. These classes mirror (but do NOT match) the ARK dataclasses.<br />
<br />
*'''Raw Item Val''' - The next question aims to establish the location of the raw item val. This is a key concept in the import tools (see notes). This question aims to establish whether a join (to a secondary table) will be necessary to arrive at this information. in the simplest case, this information is in a column on the import table itself.<br />
<br />
*'''Site Code''' - The system provides three options for the site code to be used in the import of each field. The simplest option is just to use the fixed site code specified in the CMAP. The second option is to use a code specified in a column of this table which will permit data from multiple sites to be imported from this field. Thirdly there is an option to get a site code from a joined table.<br />
<br />
In addition, the value 'notset' is available for most import types:<br />
<br />
*'''notset''' - This provides a way to specify records that should not be imported to the database. For example, you might use a keyword BLANK or SKIP in your source data as a way to prevent certain rows from being imported. Be aware that you should carefully check over the data to be imported in the extraction test.<br />
<br />
====Keys====<br />
<br />
This class is used to import the keys into the ARK database. This tool will create the itemvalues for this module. Therefore this information needs only to be imported once. Effectively this import will 'register' all of the new records to the ARK database. In ARK there are modules that use modtypes and those that don't. The import tools can import keys for modules with or without keys.<br />
<br />
*'''Modkey''' - This imports keys to a module using modtypes. It does NOT import the modtype data. It will leave this column blank in the table. The modtype information MUST be imported to avoid nasty errors.<br />
<br />
*'''Key''' - This imports keys for modules not using modtypes.<br />
<br />
The data in THIS field/column will be turned into the ARK key by appending the site code to the front of it. You need to ensure that this data would be suitable to be used as an ARK key. It must not contain underscores for example. Duplicated numbers int his column will also cause the import to crash.<br />
<br />
The form will only ask for three things:<br />
<br />
*'''UID Column''' - see notes<br />
*'''Itemkey''' - The itemkey for which these values will become ARK itemvalues (defines the module)<br />
*'''not set''' - see notes<br />
<br />
====Boolean TRUE|FALSE|unset (attribute A)====<br />
<br />
This kind of true/false data is imported to ARK as attributes. In this case it is presupposed that every row will be imported.<br />
<br />
*'''true''' - Use this to set up which values will be turned into a logical positive in ARK<br />
*'''false''' - Use this to set up which values will be turned into a logical negative in ARK<br />
<br />
Before running the import, you should make sure that the attribute itself exists.<br />
<br />
Note that in ARK booleans are represented as 0 and 1 but that this may be aliased to whatever you like.<br />
<br />
====Controlled Lists (attribute B)====<br />
<br />
Data from controlled lists (aka 'look up tables', aka 'drop down menus') will be imported into ARK as class "attribute". For import purposes (only) there are two types of attribute. Type B attributes are of the kind where a series of terms in a controlled list are in play, these may or may not be applied to the item. It is NOT presupposed that a value must be set for every item. Multiple entries per item are also permitted (each one must be on its own unique row int he source data).<br />
<br />
Before beginning the import, you must establish the attributetype of the attribute in question, if not present, this must be added before attempting to create the mapping.<br />
<br />
Any attributes themselves may be manually set up before import or will be automatically added (and aliased) by the function if they are not already present int he controlled list for the specified attribute type.<br />
<br />
Note: All boolean values for any attributes added will be forced to TRUE (1) in the import.<br />
<br />
====Numbers, Text, Dates====<br />
<br />
These three types of data are simply imported as is. The previous comments on raw itemvals and 'notset' apply<br />
<br />
====Spans====<br />
<br />
Note that you must have manually set up the span type.<br />
<br />
Spans rely on having raw data in both the beginning field and end field. This data must be pre-processed as it will be entered into ark AS IS.<br />
<br />
Only add a mapping for the 'beginning' of spans. In this mapping the column containing the 'end' data is specified as 'end_source_col'. Do not map 'ends' as this will duplicate entry<br />
<br />
====Action====<br />
<br />
At present there is no user form for adding actions to the CMAP<br />
<br />
====Links to other modules - XMI====<br />
<br />
Specify the XMI itemkey and xmi itemval column for the XMI links.<br />
<br />
===Extraction Tests and Execution===<br />
<br />
Once you have registered a mapping into the database, this can be tested. The test will do a dry run on the import and show you a table of results that show the exact values to be added to the database for this extraction from the source db. It is vital that you very closely check this data to ensure that you have got your set up correct.<br />
<br />
In addition, check for duplicates or otherwise mangled data.<br />
<br />
Undoing an incorrectly specified import will require use of the phpMyAdmin tool and will require knowledge of SQL.<br />
<br />
Bear in mind that in the case of Attributes of type B, the preview will most likely show a series of SQL statements indicating the additions to the look up table (and Aliases) that would be run.<br />
<br />
===Importing Chains===<br />
<br />
Chains can be imported by using the link in the structure defining frame: <br />
<br />
:If attributes are to be chained to this number then click <u>[here]</u>.<br />
<br />
Clicking that link will save the structure and flag it so that when it is imported the ids of the new datafrags will be inserted into a column in the import_ table named ''''%col%'''_itemval', where '''%col%''' is the name of the column to chain.<br />
<br />
[[category:administrator]]</div>Mikehttps://ark.lparchaeology.com/wiki/index.php?title=Importing_Data&diff=3345Importing Data2015-10-13T14:20:12Z<p>Mike: /* Importing Chains */</p>
<hr />
<div>As of the v1.0 release of ARK there is a set of data import tools within ARK. These tools speed up and improve the process of creating a concordance map between the source data and an ARK database. Never the less, using the tools will require a working knowledge of ARK's data structures. If you are unable to import your data or would like training in the ARK import tools, please contact [http://www.lparchaeology.com/cms/about-lp/contact L - P : Archaeology] who provide both training and custom ARK installations.<br />
<br />
==Overview==<br />
<br />
The ARK import tools are designed to provide a means to import data from table based data. This can be either in the form of a series of tables from a relational database or from a spreadsheet. The tools map the fields in the data onto fields within the ARK data structure. Data can then be inspected before running an import.<br />
<br />
The tools provide some join functionality and are intended to reduce the need for pre-processing of data where possible. Inevitably, there will be some need to pre-process data for import.<br />
<br />
===Concordance Maps===<br />
<br />
Each instance of ARK can contain multiple concordance maps. The concordance map holds the information (mappings) that explains to the import tools how a particular element of the data should be reworked for import to the ARK structure. The concordance map maps both 'structure' and 'data'.<br />
<br />
====Structure Mapping====<br />
<br />
This is the key part of each concordance map, it contains the mappings for each field in the source data that will be imported into the ARK database. Due to the way ARK data is structured, it is often the case that many source data fields are not needed for import.<br />
<br />
Once mapped, a field can be imported and re-imported using the same mapping. Future version of ARK may support this as a method for dynamic updates of the data over the web.<br />
<br />
To use a simple example, a field in the source data such as 'Description' which is a column in a source database, will be mapped to an ARK txttype ready for import.<br />
<br />
====Data====<br />
<br />
Data mapping maps certain terms or values in a source database to certain terms or values in the ARK database. For example, this permits a term such as 'castle' in a source database to be mapped to a term such as 'defensive structure' in a target database.<br />
<br />
When importing from controlled lists, ARK makes new data mappings on the fly as it finds each new term within the list.<br />
<br />
By manually creating a data mapping, it is possible to alter the way this data is imported without the need for pre-processing. This is especially important in the case of dynamically re-importing data.<br />
<br />
==How to import data==<br />
<br />
This gives an overview of how to import data into ARK using the built in import tools.<br />
<br />
===Source Data===<br />
<br />
The first task is to take a close look at the data you are about to import. Problems in the data will not be magically corrected by the import tools. If you put junk in you will get junk out.<br />
<br />
At this stage the tools will only read data in from a source database on the same MySQL server. There is no reason you couldn't adapt the tools to easily work with another database such as Postgres, but at this stage, the tools will not read in data external to the server. The first task is therefore to import your data to the server. You can either create tables on the target database or set up a separate 'source database'.<br />
<br />
The easiest way to get data onto the server is probably to save each table as a comma separated values file and import this to the server using phpMyAdmin.<br />
<br />
Be sure to prefix source database tables with "import_" this will indicate to the import tools that you wish to map data in this table. Look up tables and other supplementary tables which will not be directly imported do not need this prefix.<br />
<br />
As a rule, you should name your columns with sensible names, avoid fancy characters and so on. Try to make the names unique and memorable.<br />
<br />
It is often simplest to create the table on the database first and then simply import the data into this table (and its defined columns), although this is up to you.<br />
<br />
Once you have imported your source data to the server, it is time to start mapping your databases.<br />
<br />
*'''UID Column''' - Any table that you prefix 'import_' and intend to import data from MUST have a column which contains a unique ID for each row of the table. This is used to loop over the data by the tools, it is not imported in anyway, but it is essential. This can in fact be a column containing your 'key' data, although in certain circumstances it is desirable to create (manually) and new UID column on the source table specifically for the purpose of importing the data. See below for further information.<br />
<br />
===Concordance Map - CMAP===<br />
<br />
If you have not done so already, you will need to create a new Concordance Map. To do this, use the tool available on the left panel fo the import tools home page.<br />
<br />
Fill in the required fields carefully (check spellings!) and save the new map to the database.<br />
<br />
*'''Nickname''' - A mnemonic, this can be anything you like that will help you remember the CMAP. Do NOT use spaces or funny characters here.<br />
*'''Description''' - A text field that you can use to describe this CMAP for future reference. This will accept UTF-8 characters etc.<br />
*'''Source DB''' - The exact name of the source database on this server. (May be the same as the ARK db)<br />
*'''Target Site Code''' - This is a default site code for the import. More complex options for this can be set in the structure map (which override this setting).<br />
<br />
You can edit this map afterwards using the built in edit CMAP tool.<br />
<br />
===Structure Map - CMAP Structure===<br />
<br />
Once you have set up a CMAP, you can begin mapping fields in the source DB. The key thing here is to treat each and every field (column) in the source data s an independent entity. The import tools look only at a single column, they do not try to recreate your relational database beyond a few specific functions.<br />
<br />
The recommended method is to look at each column in turn to decide if it will be imported into ARK or not. This is not as simple as it sounds so take time to analyse each column.<br />
<br />
In order to set up each field for import, use the tools provided. On the left panel of the import pages, select 'Edit Structure Map'. The tools provided list out all the 'import_' tables available on this CMAP in the right hand panel. In order to start working simple click the edit icon next to the table.<br />
<br />
This will then display each of the fields within this table that are available for import. Fields that are already mapped will be indicated as such. Each field may only be mapped once within a given CMAP, but could be mapped from multiple CMAPs if needed.<br />
<br />
In order to create a new mapping click the edit icon for the field. The tool will ask a couple of simple questions about the field before presenting a form.<br />
<br />
*'''Class''' - The first question asks what type of import the tools should run on this data. These classes mirror (but do NOT match) the ARK dataclasses.<br />
<br />
*'''Raw Item Val''' - The next question aims to establish the location of the raw item val. This is a key concept in the import tools (see notes). This question aims to establish whether a join (to a secondary table) will be necessary to arrive at this information. in the simplest case, this information is in a column on the import table itself.<br />
<br />
*'''Site Code''' - The system provides three options for the site code to be used in the import of each field. The simplest option is just to use the fixed site code specified in the CMAP. The second option is to use a code specified in a column of this table which will permit data from multiple sites to be imported from this field. Thirdly there is an option to get a site code from a joined table.<br />
<br />
In addition, the value 'notset' is available for most import types:<br />
<br />
*'''notset''' - This provides a way to specify records that should not be imported to the database. For example, you might use a keyword BLANK or SKIP in your source data as a way to prevent certain rows from being imported. Be aware that you should carefully check over the data to be imported in the extraction test.<br />
<br />
====Keys====<br />
<br />
This class is used to import the keys into the ARK database. This tool will create the itemvalues for this module. Therefore this information needs only to be imported once. Effectively this import will 'register' all of the new records to the ARK database. In ARK there are modules that use modtypes and those that don't. The import tools can import keys for modules with or without keys.<br />
<br />
*'''Modkey''' - This imports keys to a module using modtypes. It does NOT import the modtype data. It will leave this column blank in the table. The modtype information MUST be imported to avoid nasty errors.<br />
<br />
*'''Key''' - This imports keys for modules not using modtypes.<br />
<br />
The data in THIS field/column will be turned into the ARK key by appending the site code to the front of it. You need to ensure that this data would be suitable to be used as an ARK key. It must not contain underscores for example. Duplicated numbers int his column will also cause the import to crash.<br />
<br />
The form will only ask for three things:<br />
<br />
*'''UID Column''' - see notes<br />
*'''Itemkey''' - The itemkey for which these values will become ARK itemvalues (defines the module)<br />
*'''not set''' - see notes<br />
<br />
====Boolean TRUE|FALSE|unset (attribute A)====<br />
<br />
This kind of true/false data is imported to ARK as attributes. In this case it is presupposed that every row will be imported.<br />
<br />
*'''true''' - Use this to set up which values will be turned into a logical positive in ARK<br />
*'''false''' - Use this to set up which values will be turned into a logical negative in ARK<br />
<br />
Before running the import, you should make sure that the attribute itself exists.<br />
<br />
Note that in ARK booleans are represented as 0 and 1 but that this may be aliased to whatever you like.<br />
<br />
====Controlled Lists (attribute B)====<br />
<br />
Data from controlled lists (aka 'look up tables', aka 'drop down menus') will be imported into ARK as class "attribute". For import purposes (only) there are two types of attribute. Type B attributes are of the kind where a series of terms in a controlled list are in play, these may or may not be applied to the item. It is NOT presupposed that a value must be set for every item. Multiple entries per item are also permitted (each one must be on its own unique row int he source data).<br />
<br />
Before beginning the import, you must establish the attributetype of the attribute in question, if not present, this must be added before attempting to create the mapping.<br />
<br />
Any attributes themselves may be manually set up before import or will be automatically added (and aliased) by the function if they are not already present int he controlled list for the specified attribute type.<br />
<br />
Note: All boolean values for any attributes added will be forced to TRUE (1) in the import.<br />
<br />
====Numbers, Text, Dates====<br />
<br />
These three types of data are simply imported as is. The previous comments on raw itemvals and 'notset' apply<br />
<br />
====Spans====<br />
<br />
Note that you must have manually set up the span type.<br />
<br />
Spans rely on having raw data in both the beginning field and end field. This data must be pre-processed as it will be entered into ark AS IS.<br />
<br />
Only add a mapping for the 'beginning' of spans. In this mapping the column containing the 'end' data is specified as 'end_source_col'. Do not map 'ends' as this will duplicate entry<br />
<br />
====Action====<br />
<br />
At present there is no user form for adding actions to the CMAP<br />
<br />
====Links to other modules - XMI====<br />
<br />
Specify the XMI itemkey and xmi itemval column for the XMI links.<br />
<br />
===Extraction Tests and Execution===<br />
<br />
Once you have registered a mapping into the database, this can be tested. The test will do a dry run on the import and show you a table of results that show the exact values to be added to the database for this extraction from the source db. It is vital that you very closely check this data to ensure that you have got your set up correct.<br />
<br />
In addition, check for duplicates or otherwise mangled data.<br />
<br />
Undoing an incorrectly specified import will require use of the phpMyAdmin tool and will require knowledge of SQL.<br />
<br />
Bear in mind that in the case of Attributes of type B, the preview will most likely show a series of SQL statements indicating the additions to the look up table (and Aliases) that would be run.<br />
<br />
===Importing Chains===<br />
<br />
Chains can be imported by using the link in the structure defining frame: <br />
<br />
If attributes are to be chained to this number then click [here].<br />
<br />
Clicking that link will save the structure and flag it so that when it is imported the ids of the new datafrags will be inserted into a column in the import_ table named ''''%col%'''_itemval', where '''%col%''' is the name of the column to chain.<br />
<br />
[[category:administrator]]</div>Mikehttps://ark.lparchaeology.com/wiki/index.php?title=Importing_Data&diff=3344Importing Data2015-10-13T14:19:48Z<p>Mike: /* Importing Chains */</p>
<hr />
<div>As of the v1.0 release of ARK there is a set of data import tools within ARK. These tools speed up and improve the process of creating a concordance map between the source data and an ARK database. Never the less, using the tools will require a working knowledge of ARK's data structures. If you are unable to import your data or would like training in the ARK import tools, please contact [http://www.lparchaeology.com/cms/about-lp/contact L - P : Archaeology] who provide both training and custom ARK installations.<br />
<br />
==Overview==<br />
<br />
The ARK import tools are designed to provide a means to import data from table based data. This can be either in the form of a series of tables from a relational database or from a spreadsheet. The tools map the fields in the data onto fields within the ARK data structure. Data can then be inspected before running an import.<br />
<br />
The tools provide some join functionality and are intended to reduce the need for pre-processing of data where possible. Inevitably, there will be some need to pre-process data for import.<br />
<br />
===Concordance Maps===<br />
<br />
Each instance of ARK can contain multiple concordance maps. The concordance map holds the information (mappings) that explains to the import tools how a particular element of the data should be reworked for import to the ARK structure. The concordance map maps both 'structure' and 'data'.<br />
<br />
====Structure Mapping====<br />
<br />
This is the key part of each concordance map, it contains the mappings for each field in the source data that will be imported into the ARK database. Due to the way ARK data is structured, it is often the case that many source data fields are not needed for import.<br />
<br />
Once mapped, a field can be imported and re-imported using the same mapping. Future version of ARK may support this as a method for dynamic updates of the data over the web.<br />
<br />
To use a simple example, a field in the source data such as 'Description' which is a column in a source database, will be mapped to an ARK txttype ready for import.<br />
<br />
====Data====<br />
<br />
Data mapping maps certain terms or values in a source database to certain terms or values in the ARK database. For example, this permits a term such as 'castle' in a source database to be mapped to a term such as 'defensive structure' in a target database.<br />
<br />
When importing from controlled lists, ARK makes new data mappings on the fly as it finds each new term within the list.<br />
<br />
By manually creating a data mapping, it is possible to alter the way this data is imported without the need for pre-processing. This is especially important in the case of dynamically re-importing data.<br />
<br />
==How to import data==<br />
<br />
This gives an overview of how to import data into ARK using the built in import tools.<br />
<br />
===Source Data===<br />
<br />
The first task is to take a close look at the data you are about to import. Problems in the data will not be magically corrected by the import tools. If you put junk in you will get junk out.<br />
<br />
At this stage the tools will only read data in from a source database on the same MySQL server. There is no reason you couldn't adapt the tools to easily work with another database such as Postgres, but at this stage, the tools will not read in data external to the server. The first task is therefore to import your data to the server. You can either create tables on the target database or set up a separate 'source database'.<br />
<br />
The easiest way to get data onto the server is probably to save each table as a comma separated values file and import this to the server using phpMyAdmin.<br />
<br />
Be sure to prefix source database tables with "import_" this will indicate to the import tools that you wish to map data in this table. Look up tables and other supplementary tables which will not be directly imported do not need this prefix.<br />
<br />
As a rule, you should name your columns with sensible names, avoid fancy characters and so on. Try to make the names unique and memorable.<br />
<br />
It is often simplest to create the table on the database first and then simply import the data into this table (and its defined columns), although this is up to you.<br />
<br />
Once you have imported your source data to the server, it is time to start mapping your databases.<br />
<br />
*'''UID Column''' - Any table that you prefix 'import_' and intend to import data from MUST have a column which contains a unique ID for each row of the table. This is used to loop over the data by the tools, it is not imported in anyway, but it is essential. This can in fact be a column containing your 'key' data, although in certain circumstances it is desirable to create (manually) and new UID column on the source table specifically for the purpose of importing the data. See below for further information.<br />
<br />
===Concordance Map - CMAP===<br />
<br />
If you have not done so already, you will need to create a new Concordance Map. To do this, use the tool available on the left panel fo the import tools home page.<br />
<br />
Fill in the required fields carefully (check spellings!) and save the new map to the database.<br />
<br />
*'''Nickname''' - A mnemonic, this can be anything you like that will help you remember the CMAP. Do NOT use spaces or funny characters here.<br />
*'''Description''' - A text field that you can use to describe this CMAP for future reference. This will accept UTF-8 characters etc.<br />
*'''Source DB''' - The exact name of the source database on this server. (May be the same as the ARK db)<br />
*'''Target Site Code''' - This is a default site code for the import. More complex options for this can be set in the structure map (which override this setting).<br />
<br />
You can edit this map afterwards using the built in edit CMAP tool.<br />
<br />
===Structure Map - CMAP Structure===<br />
<br />
Once you have set up a CMAP, you can begin mapping fields in the source DB. The key thing here is to treat each and every field (column) in the source data s an independent entity. The import tools look only at a single column, they do not try to recreate your relational database beyond a few specific functions.<br />
<br />
The recommended method is to look at each column in turn to decide if it will be imported into ARK or not. This is not as simple as it sounds so take time to analyse each column.<br />
<br />
In order to set up each field for import, use the tools provided. On the left panel of the import pages, select 'Edit Structure Map'. The tools provided list out all the 'import_' tables available on this CMAP in the right hand panel. In order to start working simple click the edit icon next to the table.<br />
<br />
This will then display each of the fields within this table that are available for import. Fields that are already mapped will be indicated as such. Each field may only be mapped once within a given CMAP, but could be mapped from multiple CMAPs if needed.<br />
<br />
In order to create a new mapping click the edit icon for the field. The tool will ask a couple of simple questions about the field before presenting a form.<br />
<br />
*'''Class''' - The first question asks what type of import the tools should run on this data. These classes mirror (but do NOT match) the ARK dataclasses.<br />
<br />
*'''Raw Item Val''' - The next question aims to establish the location of the raw item val. This is a key concept in the import tools (see notes). This question aims to establish whether a join (to a secondary table) will be necessary to arrive at this information. in the simplest case, this information is in a column on the import table itself.<br />
<br />
*'''Site Code''' - The system provides three options for the site code to be used in the import of each field. The simplest option is just to use the fixed site code specified in the CMAP. The second option is to use a code specified in a column of this table which will permit data from multiple sites to be imported from this field. Thirdly there is an option to get a site code from a joined table.<br />
<br />
In addition, the value 'notset' is available for most import types:<br />
<br />
*'''notset''' - This provides a way to specify records that should not be imported to the database. For example, you might use a keyword BLANK or SKIP in your source data as a way to prevent certain rows from being imported. Be aware that you should carefully check over the data to be imported in the extraction test.<br />
<br />
====Keys====<br />
<br />
This class is used to import the keys into the ARK database. This tool will create the itemvalues for this module. Therefore this information needs only to be imported once. Effectively this import will 'register' all of the new records to the ARK database. In ARK there are modules that use modtypes and those that don't. The import tools can import keys for modules with or without keys.<br />
<br />
*'''Modkey''' - This imports keys to a module using modtypes. It does NOT import the modtype data. It will leave this column blank in the table. The modtype information MUST be imported to avoid nasty errors.<br />
<br />
*'''Key''' - This imports keys for modules not using modtypes.<br />
<br />
The data in THIS field/column will be turned into the ARK key by appending the site code to the front of it. You need to ensure that this data would be suitable to be used as an ARK key. It must not contain underscores for example. Duplicated numbers int his column will also cause the import to crash.<br />
<br />
The form will only ask for three things:<br />
<br />
*'''UID Column''' - see notes<br />
*'''Itemkey''' - The itemkey for which these values will become ARK itemvalues (defines the module)<br />
*'''not set''' - see notes<br />
<br />
====Boolean TRUE|FALSE|unset (attribute A)====<br />
<br />
This kind of true/false data is imported to ARK as attributes. In this case it is presupposed that every row will be imported.<br />
<br />
*'''true''' - Use this to set up which values will be turned into a logical positive in ARK<br />
*'''false''' - Use this to set up which values will be turned into a logical negative in ARK<br />
<br />
Before running the import, you should make sure that the attribute itself exists.<br />
<br />
Note that in ARK booleans are represented as 0 and 1 but that this may be aliased to whatever you like.<br />
<br />
====Controlled Lists (attribute B)====<br />
<br />
Data from controlled lists (aka 'look up tables', aka 'drop down menus') will be imported into ARK as class "attribute". For import purposes (only) there are two types of attribute. Type B attributes are of the kind where a series of terms in a controlled list are in play, these may or may not be applied to the item. It is NOT presupposed that a value must be set for every item. Multiple entries per item are also permitted (each one must be on its own unique row int he source data).<br />
<br />
Before beginning the import, you must establish the attributetype of the attribute in question, if not present, this must be added before attempting to create the mapping.<br />
<br />
Any attributes themselves may be manually set up before import or will be automatically added (and aliased) by the function if they are not already present int he controlled list for the specified attribute type.<br />
<br />
Note: All boolean values for any attributes added will be forced to TRUE (1) in the import.<br />
<br />
====Numbers, Text, Dates====<br />
<br />
These three types of data are simply imported as is. The previous comments on raw itemvals and 'notset' apply<br />
<br />
====Spans====<br />
<br />
Note that you must have manually set up the span type.<br />
<br />
Spans rely on having raw data in both the beginning field and end field. This data must be pre-processed as it will be entered into ark AS IS.<br />
<br />
Only add a mapping for the 'beginning' of spans. In this mapping the column containing the 'end' data is specified as 'end_source_col'. Do not map 'ends' as this will duplicate entry<br />
<br />
====Action====<br />
<br />
At present there is no user form for adding actions to the CMAP<br />
<br />
====Links to other modules - XMI====<br />
<br />
Specify the XMI itemkey and xmi itemval column for the XMI links.<br />
<br />
===Extraction Tests and Execution===<br />
<br />
Once you have registered a mapping into the database, this can be tested. The test will do a dry run on the import and show you a table of results that show the exact values to be added to the database for this extraction from the source db. It is vital that you very closely check this data to ensure that you have got your set up correct.<br />
<br />
In addition, check for duplicates or otherwise mangled data.<br />
<br />
Undoing an incorrectly specified import will require use of the phpMyAdmin tool and will require knowledge of SQL.<br />
<br />
Bear in mind that in the case of Attributes of type B, the preview will most likely show a series of SQL statements indicating the additions to the look up table (and Aliases) that would be run.<br />
<br />
===Importing Chains===<br />
<br />
Chains can be imported by using the link in the structure defining frame: <br />
<br />
If attributes are to be chained to this number then click [here].<br />
<br />
Clicking that link will save the structure and flag it so that when it is imported the ids of the new datafrags will be inserted into a column in the import_ table named ''''%col%'''_itemval', where '''%col%''' is the name of the column to chain<br />
<br />
[[category:administrator]]</div>Mikehttps://ark.lparchaeology.com/wiki/index.php?title=Importing_Data&diff=3343Importing Data2015-10-13T14:18:41Z<p>Mike: </p>
<hr />
<div>As of the v1.0 release of ARK there is a set of data import tools within ARK. These tools speed up and improve the process of creating a concordance map between the source data and an ARK database. Never the less, using the tools will require a working knowledge of ARK's data structures. If you are unable to import your data or would like training in the ARK import tools, please contact [http://www.lparchaeology.com/cms/about-lp/contact L - P : Archaeology] who provide both training and custom ARK installations.<br />
<br />
==Overview==<br />
<br />
The ARK import tools are designed to provide a means to import data from table based data. This can be either in the form of a series of tables from a relational database or from a spreadsheet. The tools map the fields in the data onto fields within the ARK data structure. Data can then be inspected before running an import.<br />
<br />
The tools provide some join functionality and are intended to reduce the need for pre-processing of data where possible. Inevitably, there will be some need to pre-process data for import.<br />
<br />
===Concordance Maps===<br />
<br />
Each instance of ARK can contain multiple concordance maps. The concordance map holds the information (mappings) that explains to the import tools how a particular element of the data should be reworked for import to the ARK structure. The concordance map maps both 'structure' and 'data'.<br />
<br />
====Structure Mapping====<br />
<br />
This is the key part of each concordance map, it contains the mappings for each field in the source data that will be imported into the ARK database. Due to the way ARK data is structured, it is often the case that many source data fields are not needed for import.<br />
<br />
Once mapped, a field can be imported and re-imported using the same mapping. Future version of ARK may support this as a method for dynamic updates of the data over the web.<br />
<br />
To use a simple example, a field in the source data such as 'Description' which is a column in a source database, will be mapped to an ARK txttype ready for import.<br />
<br />
====Data====<br />
<br />
Data mapping maps certain terms or values in a source database to certain terms or values in the ARK database. For example, this permits a term such as 'castle' in a source database to be mapped to a term such as 'defensive structure' in a target database.<br />
<br />
When importing from controlled lists, ARK makes new data mappings on the fly as it finds each new term within the list.<br />
<br />
By manually creating a data mapping, it is possible to alter the way this data is imported without the need for pre-processing. This is especially important in the case of dynamically re-importing data.<br />
<br />
==How to import data==<br />
<br />
This gives an overview of how to import data into ARK using the built in import tools.<br />
<br />
===Source Data===<br />
<br />
The first task is to take a close look at the data you are about to import. Problems in the data will not be magically corrected by the import tools. If you put junk in you will get junk out.<br />
<br />
At this stage the tools will only read data in from a source database on the same MySQL server. There is no reason you couldn't adapt the tools to easily work with another database such as Postgres, but at this stage, the tools will not read in data external to the server. The first task is therefore to import your data to the server. You can either create tables on the target database or set up a separate 'source database'.<br />
<br />
The easiest way to get data onto the server is probably to save each table as a comma separated values file and import this to the server using phpMyAdmin.<br />
<br />
Be sure to prefix source database tables with "import_" this will indicate to the import tools that you wish to map data in this table. Look up tables and other supplementary tables which will not be directly imported do not need this prefix.<br />
<br />
As a rule, you should name your columns with sensible names, avoid fancy characters and so on. Try to make the names unique and memorable.<br />
<br />
It is often simplest to create the table on the database first and then simply import the data into this table (and its defined columns), although this is up to you.<br />
<br />
Once you have imported your source data to the server, it is time to start mapping your databases.<br />
<br />
*'''UID Column''' - Any table that you prefix 'import_' and intend to import data from MUST have a column which contains a unique ID for each row of the table. This is used to loop over the data by the tools, it is not imported in anyway, but it is essential. This can in fact be a column containing your 'key' data, although in certain circumstances it is desirable to create (manually) and new UID column on the source table specifically for the purpose of importing the data. See below for further information.<br />
<br />
===Concordance Map - CMAP===<br />
<br />
If you have not done so already, you will need to create a new Concordance Map. To do this, use the tool available on the left panel fo the import tools home page.<br />
<br />
Fill in the required fields carefully (check spellings!) and save the new map to the database.<br />
<br />
*'''Nickname''' - A mnemonic, this can be anything you like that will help you remember the CMAP. Do NOT use spaces or funny characters here.<br />
*'''Description''' - A text field that you can use to describe this CMAP for future reference. This will accept UTF-8 characters etc.<br />
*'''Source DB''' - The exact name of the source database on this server. (May be the same as the ARK db)<br />
*'''Target Site Code''' - This is a default site code for the import. More complex options for this can be set in the structure map (which override this setting).<br />
<br />
You can edit this map afterwards using the built in edit CMAP tool.<br />
<br />
===Structure Map - CMAP Structure===<br />
<br />
Once you have set up a CMAP, you can begin mapping fields in the source DB. The key thing here is to treat each and every field (column) in the source data s an independent entity. The import tools look only at a single column, they do not try to recreate your relational database beyond a few specific functions.<br />
<br />
The recommended method is to look at each column in turn to decide if it will be imported into ARK or not. This is not as simple as it sounds so take time to analyse each column.<br />
<br />
In order to set up each field for import, use the tools provided. On the left panel of the import pages, select 'Edit Structure Map'. The tools provided list out all the 'import_' tables available on this CMAP in the right hand panel. In order to start working simple click the edit icon next to the table.<br />
<br />
This will then display each of the fields within this table that are available for import. Fields that are already mapped will be indicated as such. Each field may only be mapped once within a given CMAP, but could be mapped from multiple CMAPs if needed.<br />
<br />
In order to create a new mapping click the edit icon for the field. The tool will ask a couple of simple questions about the field before presenting a form.<br />
<br />
*'''Class''' - The first question asks what type of import the tools should run on this data. These classes mirror (but do NOT match) the ARK dataclasses.<br />
<br />
*'''Raw Item Val''' - The next question aims to establish the location of the raw item val. This is a key concept in the import tools (see notes). This question aims to establish whether a join (to a secondary table) will be necessary to arrive at this information. in the simplest case, this information is in a column on the import table itself.<br />
<br />
*'''Site Code''' - The system provides three options for the site code to be used in the import of each field. The simplest option is just to use the fixed site code specified in the CMAP. The second option is to use a code specified in a column of this table which will permit data from multiple sites to be imported from this field. Thirdly there is an option to get a site code from a joined table.<br />
<br />
In addition, the value 'notset' is available for most import types:<br />
<br />
*'''notset''' - This provides a way to specify records that should not be imported to the database. For example, you might use a keyword BLANK or SKIP in your source data as a way to prevent certain rows from being imported. Be aware that you should carefully check over the data to be imported in the extraction test.<br />
<br />
====Keys====<br />
<br />
This class is used to import the keys into the ARK database. This tool will create the itemvalues for this module. Therefore this information needs only to be imported once. Effectively this import will 'register' all of the new records to the ARK database. In ARK there are modules that use modtypes and those that don't. The import tools can import keys for modules with or without keys.<br />
<br />
*'''Modkey''' - This imports keys to a module using modtypes. It does NOT import the modtype data. It will leave this column blank in the table. The modtype information MUST be imported to avoid nasty errors.<br />
<br />
*'''Key''' - This imports keys for modules not using modtypes.<br />
<br />
The data in THIS field/column will be turned into the ARK key by appending the site code to the front of it. You need to ensure that this data would be suitable to be used as an ARK key. It must not contain underscores for example. Duplicated numbers int his column will also cause the import to crash.<br />
<br />
The form will only ask for three things:<br />
<br />
*'''UID Column''' - see notes<br />
*'''Itemkey''' - The itemkey for which these values will become ARK itemvalues (defines the module)<br />
*'''not set''' - see notes<br />
<br />
====Boolean TRUE|FALSE|unset (attribute A)====<br />
<br />
This kind of true/false data is imported to ARK as attributes. In this case it is presupposed that every row will be imported.<br />
<br />
*'''true''' - Use this to set up which values will be turned into a logical positive in ARK<br />
*'''false''' - Use this to set up which values will be turned into a logical negative in ARK<br />
<br />
Before running the import, you should make sure that the attribute itself exists.<br />
<br />
Note that in ARK booleans are represented as 0 and 1 but that this may be aliased to whatever you like.<br />
<br />
====Controlled Lists (attribute B)====<br />
<br />
Data from controlled lists (aka 'look up tables', aka 'drop down menus') will be imported into ARK as class "attribute". For import purposes (only) there are two types of attribute. Type B attributes are of the kind where a series of terms in a controlled list are in play, these may or may not be applied to the item. It is NOT presupposed that a value must be set for every item. Multiple entries per item are also permitted (each one must be on its own unique row int he source data).<br />
<br />
Before beginning the import, you must establish the attributetype of the attribute in question, if not present, this must be added before attempting to create the mapping.<br />
<br />
Any attributes themselves may be manually set up before import or will be automatically added (and aliased) by the function if they are not already present int he controlled list for the specified attribute type.<br />
<br />
Note: All boolean values for any attributes added will be forced to TRUE (1) in the import.<br />
<br />
====Numbers, Text, Dates====<br />
<br />
These three types of data are simply imported as is. The previous comments on raw itemvals and 'notset' apply<br />
<br />
====Spans====<br />
<br />
Note that you must have manually set up the span type.<br />
<br />
Spans rely on having raw data in both the beginning field and end field. This data must be pre-processed as it will be entered into ark AS IS.<br />
<br />
Only add a mapping for the 'beginning' of spans. In this mapping the column containing the 'end' data is specified as 'end_source_col'. Do not map 'ends' as this will duplicate entry<br />
<br />
====Action====<br />
<br />
At present there is no user form for adding actions to the CMAP<br />
<br />
====Links to other modules - XMI====<br />
<br />
Specify the XMI itemkey and xmi itemval column for the XMI links.<br />
<br />
===Extraction Tests and Execution===<br />
<br />
Once you have registered a mapping into the database, this can be tested. The test will do a dry run on the import and show you a table of results that show the exact values to be added to the database for this extraction from the source db. It is vital that you very closely check this data to ensure that you have got your set up correct.<br />
<br />
In addition, check for duplicates or otherwise mangled data.<br />
<br />
Undoing an incorrectly specified import will require use of the phpMyAdmin tool and will require knowledge of SQL.<br />
<br />
Bear in mind that in the case of Attributes of type B, the preview will most likely show a series of SQL statements indicating the additions to the look up table (and Aliases) that would be run.<br />
<br />
===Importing Chains===<br />
<br />
Chains can be imported by using the link in the structure defining frame: <br />
<br />
If attributes are to be chained to this number then click [here].<br />
<br />
Clicking that link will save the structure and flag it so that when it is imported the ids of the new datafrags will be inserted into a column in the import_ table named '[b]%col%[/b]_itemval', where [b]%col%[b] is the name of the column to chain<br />
<br />
[[category:administrator]]</div>Mikehttps://ark.lparchaeology.com/wiki/index.php?title=Sf_chaintable&diff=3342Sf chaintable2015-10-13T12:41:35Z<p>Mike: /* Description */</p>
<hr />
<div>===Description===<br />
<br />
sf_chaintable can be used for displaying and editing data fragments that are chained to other data fragments. One 'root' field is linked to the item being viewed, which may have one or more fragments per item creating rows for the table and then one or more fragments can be chained to these to create additional columns. Examples of this include attaching numbers to find attributes - mirroring and supplements [[sf_assemblage]] for linking usage to periods of sites, or for levels tables.<br />
<br />
By default the chaintable hides any empty columns. As of version 1.2 the new records feature is incomplete and only some of the edit functions are supported - numbers, text and attributes.<br />
<br />
===Additional Fields===<br />
<br />
The sf_assemblage needs a number of different 'op' variables set in the sf_conf:<br />
*'''op_assemblage_type''' - the root field which is connected to the item<br />
*'''op_sum_field''' - if this is set then the rows will be aggregated where they are identical and the number in this field will be summed for the output. this field must be a number!<br />
*'''fields''' - the columns that will be added to the table<br />
<br />
===Example Configuration===<br />
<pre><br />
$conf_mcd_assemblage =<br />
array(<br />
'view_state' => 'max',<br />
'edit_state' => 'view',<br />
'sf_title' => 'assemblage', //appears in the titlebar of the subform (mk nname)<br />
'sf_html_id' => 'pottery_ass', //the form id tag (must be unique)<br />
'sf_nav_type' => 'nmedit',<br />
'script' => 'php/subforms/sf_chaintable.php',<br />
'op_assemblage_type' => $conf_field_total,// the field of data attached to the item<br />
'op_sum_field' => $conf_field_total,<br />
'default' => '0', //the default value for creating new records<br />
'op_label' => 'space',<br />
'op_input' => 'save',<br />
'fields' => array(<br />
$conf_field_sherd_class,<br />
$conf_field_sherd_form,<br />
$conf_field_sherd_type,<br />
$conf_field_finddates,<br />
$conf_field_sherdfunction,<br />
$conf_field_catxmiloc,<br />
$conf_field_total,<br />
),<br />
);<br />
</pre><br />
<br />
<br />
[[category: Administrator]]</div>Mikehttps://ark.lparchaeology.com/wiki/index.php?title=Sf_chaintable&diff=3341Sf chaintable2015-10-12T16:52:49Z<p>Mike: </p>
<hr />
<div>===Description===<br />
<br />
sf_chaintable can be used for displaying and editing data fragments that are chained to other data fragments. One 'root' field is linked to the item being viewed, which may have one or more fragments per item creating rows for the table and then one or more fragments can be chained to these to create additional columns. Examples of this include attaching numbers to find attributes - mirroring and supplements [[sf_assemblage]] for linking usage to periods of sites, or for levels tables.<br />
<br />
By default the chaintable hides any empty columns. As of version 1.2 the new records feature is uncomplete and only some of the edit functions are supported - numbers, text and attributes.<br />
<br />
===Additional Fields===<br />
<br />
The sf_assemblage needs a number of different 'op' variables set in the sf_conf:<br />
*'''op_assemblage_type''' - the root field which is connected to the item<br />
*'''op_sum_field''' - if this is set then the rows will be aggregated where they are identical and the number in this field will be summed for the output. this field must be a number!<br />
*'''fields''' - the columns that will be added to the table<br />
<br />
===Example Configuration===<br />
<pre><br />
$conf_mcd_assemblage =<br />
array(<br />
'view_state' => 'max',<br />
'edit_state' => 'view',<br />
'sf_title' => 'assemblage', //appears in the titlebar of the subform (mk nname)<br />
'sf_html_id' => 'pottery_ass', //the form id tag (must be unique)<br />
'sf_nav_type' => 'nmedit',<br />
'script' => 'php/subforms/sf_chaintable.php',<br />
'op_assemblage_type' => $conf_field_total,// the field of data attached to the item<br />
'op_sum_field' => $conf_field_total,<br />
'default' => '0', //the default value for creating new records<br />
'op_label' => 'space',<br />
'op_input' => 'save',<br />
'fields' => array(<br />
$conf_field_sherd_class,<br />
$conf_field_sherd_form,<br />
$conf_field_sherd_type,<br />
$conf_field_finddates,<br />
$conf_field_sherdfunction,<br />
$conf_field_catxmiloc,<br />
$conf_field_total,<br />
),<br />
);<br />
</pre><br />
<br />
<br />
[[category: Administrator]]</div>Mikehttps://ark.lparchaeology.com/wiki/index.php?title=Sf_chaintable&diff=3340Sf chaintable2015-10-12T16:24:37Z<p>Mike: Created page with "Example configuration: $conf_mcd_assemblage = array( 'view_state' => 'max', 'edit_state' => 'view', 'sf_title' => 'assemblage', //appears in the titl..."</p>
<hr />
<div>Example configuration:<br />
<br />
$conf_mcd_assemblage =<br />
array(<br />
'view_state' => 'max',<br />
'edit_state' => 'view',<br />
'sf_title' => 'assemblage', //appears in the titlebar of the subform (mk nname)<br />
'sf_html_id' => 'pottery_ass', //the form id tag (must be unique)<br />
'sf_nav_type' => 'nmedit',<br />
'script' => 'php/subforms/sf_chaintable.php',<br />
'op_assemblage_type' => $conf_field_total,// the field of data attached to the item<br />
'op_sum_field' => $conf_field_total,<br />
'default' => '0', //the default value for creating new records<br />
// 'op_chart' => 'yes',<br />
'op_label' => 'space',<br />
'op_input' => 'save',<br />
'fields' => array(<br />
$conf_field_sherd_class,<br />
$conf_field_sherd_form,<br />
$conf_field_sherd_type,<br />
$conf_field_finddates,<br />
$conf_field_sherdfunction,<br />
//$conf_field_locxmicat,<br />
$conf_field_catxmiloc,<br />
$conf_field_total,<br />
),<br />
);</div>Mikehttps://ark.lparchaeology.com/wiki/index.php?title=Subform_Reference&diff=3339Subform Reference2015-10-12T16:23:10Z<p>Mike: /* Complex Subforms */</p>
<hr />
<div>The following is a list of all current ARK subforms. Each subform page includes detailed configuration information and an example subform setup.<br />
<br />
'''NB - AS OF ARK 0.7 THE CONFIGURATION OF SUBFORMS USING MODTYPES WILL CHANGE TO RELY SOLELY ON THE [[Conditional Subforms]] OPTION.'''<br />
<br />
===Text Subforms===<br />
*'''[[sf_txt]]''' - subform to enter text data<br />
<br />
===Action Subforms===<br />
*'''[[sf_action]]''' - subform to link actions (actors with roles) to items<br />
<br />
===Number Subforms===<br />
*'''[[sf_number]]''' - subform to add number data<br />
*'''[[sf_geo_number]]''' - subform to add easting and northings to an item - and then automatically create a GML file for the item<br />
<br />
===Attribute Subforms===<br />
*'''[[sf_finds]]''' - subform to attach attributes of a given type (findtypes) and to set a boolean for each one<br />
*'''[[sf_attr_bytype]]''' - subform to attach attributes of a given type<br />
*'''[[sf_attr_boolean]]''' - subform to set boolean attribute<br />
<br />
===Spatial Subforms===<br />
*'''[[sf_spat]]''' - a subform to display spatial data about an item ''THIS SUBFORM IS DEPRECATED AND SHOULD NOT BE USED FOR ANY NEW ARK SYSTEMS''<br />
*'''[[sf_wfs_spat]]''' - the new subform to display spatial data about an item<br />
*'''[[sf_place]]''' - a subform for displaying places that item's geometry intersects with<br />
<br />
===Span Subforms===<br />
*'''[[sf_span_rel]]''' - subform to enter span relationships not in matrix form<br />
*'''[[sf_spanmatrix]]''' - subform to enter span relationships in matrix form<br />
<br />
===File Subforms===<br />
*'''[[sf_file]]''' - subform to handle files<br />
*'''[[sf_mediabrowser]]''' - subform to browse and upload media such as images or videos<br />
<br />
===XMI Subforms===<br />
*'''[[sf_xmi]]''' - subform to enter XMI links between records<br />
<br />
===Complex Subforms===<br />
Complex subforms combine different data classes.<br />
<br />
*'''[[sf_interp]]''' - subform to handle interpretative text fragments with actor/date pairings<br />
*'''[[sf_assemblage]]''' - a subform to handle bulk assemblages<br />
*'''[[sf_event]]''' - subform to handle actor/date pairs<br />
*'''[[sf_frame]]''' - subform to display many subforms within a single subform<br />
*'''[[sf_chaintable]]''' - subform to display tables made from chains<br />
<br />
===Item/Record Subforms===<br />
Subforms used for working on an item record itself.<br />
<br />
*'''[[sf_modtype]]''' - subform to handle modules with different item types<br />
*'''[[sf_delete_record]]''' - subform to handle modules with different item types<br />
*'''[[sf_itemval]]''' - subform to handle modules with different item types<br />
<br />
The following subforms offer conflict resolution feedback to users and work directly with the above subforms<br />
<br />
*'''[[sf_dnarecord]]''' - subform to handle modules with different item types<br />
*'''[[sf_modtype_conflicts]]''' - subform to handle modules with different item types<br />
<br />
===Other Subforms===<br />
Other subforms are forms that don't really fit anywhere else.<br />
<br />
*'''[[sf_flickr]]''' - subform to dynamically pull photos from a flickr stream<br />
*'''[[sf_websites]]''' - subform to generate a list of websites that can take dynamic variables (for instance a link to youTube searching all of the videos according to a value held in ARK)<br />
*'''[[sf_rssfeed]]''' - subform to pull the results from one or more RSS feeds (including feeds from ARK itself)<br />
*'''[[sf_qrtag]]''' - subform to create a QR tag for an item - to be used from smart phones/QR tag readers<br />
*'''[[sf_ctrl_lst]]''' - a pseudo subform to allow users to add to controlled lists<br />
*'''[[sf_geonames]]''' - subform to pull results from the GeoNames API for display<br />
<br />
===Non Data Subforms===<br />
Subforms that are not used to handle data entry or viewing. These are typically used to make custom dynamic content within panels and home pages. The code is reusable and transferable. In addition these make use of conditional behaviours.<br />
<br />
*'''[[sf_linklist]]''' - a subform that spits out a formatted list of links with dynamically marked up anchors<br />
<br />
[[category:administrator]]</div>Mikehttps://ark.lparchaeology.com/wiki/index.php?title=Wordpress_ark_plugin&diff=3338Wordpress ark plugin2015-10-05T17:10:30Z<p>Mike: Redirected page to Setting up the Wordpress Plugin</p>
<hr />
<div>#REDIRECT [[Setting_up_the_Wordpress_Plugin]]</div>Mikehttps://ark.lparchaeology.com/wiki/index.php?title=Wordpress_ark_plugin&diff=3337Wordpress ark plugin2015-10-05T17:10:00Z<p>Mike: Redirected page to Setting up Wordpress</p>
<hr />
<div>#REDIRECT [[Setting up Wordpress]]</div>Mikehttps://ark.lparchaeology.com/wiki/index.php?title=Importing_Text_Data&diff=3336Importing Text Data2015-09-14T11:38:04Z<p>Mike: /* Step 4. Test Import */</p>
<hr />
<div>==ARK textfile importer user guide==<br />
Since v1.1.2 ARK ahs had a tool to import text based files. in either JSON or CSV <br />
Data represented in either json or csv format can be uploaded to the ARK database.The file can be filtered so that only certain records are uploaded. It can create new records or add to existing ones.<br />
<br />
===Step 1. Identify datafile===<br />
Specify the location of the file.The import page of ARK in 1.1.2 includes a function for importing data directly from a text file or a data stream available online. Currently the file must be available on a web server. (Fig. 1) Enter the location of the file, either a relative path from the ARK root eg data/uploads/your_data.csv or a remote web address eg http://www.example.com/ARK/data?format=json.<br />
Click Submit.<br />
<br />
Depend on the size of the file it may take some time to load into your browser. Once it is loaded it will be available to manipulate very quickly.A spinner will appear while this is happening.<br />
<br />
===Step 2. Determine 'Root' of the data structure.===<br />
<br />
[[File:jsonhowto_fig1.png|thumb|400px|right|Figure 1: File Picker]]<br />
<br />
The data you have will likely include information that relates to several ARK items. Within your data structure there will be a level that includes all of these item representations - in the same way as rows in a table. In a csv file where there is a single record per row the root will be 'root>items'. In a more complicated json file (for example one with file level metadata held in the root as well as the items) the records may be contained in an object within the root object. Json allows for extensible representation, so each 'row' in the json file may not include all the same fields. It may be necessary to use the filter tools to only import objects which have a certain criteria.<br />
<br />
The example shown here has many items in the items object in root. The first available row is used to identify each of the contained objects - in this case it is an ARK ID, but it could be anything and may not be unique if it is not unique in your structure. For example if field 1 in your table is a site code or context type.<br />
<br />
[[File:jsonhowto_fig2.png|thumb|400px|left|none|Figure 2: Objects in 'root>items']]<br />
<br />
In order to define this level you will need to navigate to it.The path at the top of the page describes where in the data structure you currently are.The large panel below the advanced options shows the identifiers held in the current level.At the root level of a CSV file the first column will be used as an identifier. Clicking on these will open the data attached to that identifier for viewing.(Fig. 2) It is possible to go up one level using the button at the end of the list of objects, or any part of the location at the top of the page can be used to navigate to that level.When the main window would contain more than 20 items it is truncated and only the first 20 are shown.<br />
<br />
Use the 'this is root' button in the navigation panel at the top of the page when the root is displayed in the main panel.<br />
<br />
===Step 3. Determine ARK IDs===<br />
<br />
[[File:jsonhowto_fig3.png|thumb|right|600px|Figure 3: Root and ARK ID specified]]<br />
<br />
Using the navigation method above find the path to the ARK id within your data. This will often be as simple as 'root>items>item1>Ark_id'.The object that you define the ARK ID in is not important. As long as the objects are structured consistently the ARK IDs will be retrieved automatically based on the root defined in the step above. so for instance the importer will import the ARK ID for METSUR_152 from root > items > METSUR_152 > ARK_id and then the ARK ID for METSUR_153 from root > items > METSUR_153 > ARK_id, and so on.<br />
<br />
<br />
If you do not have ARK IDs in your data refer to the advanced options.<br />
Otherwise, click 'This is ARK ID' button below the identifier that contains the ARK ID.<br />
<br />
===Step 4. Test Import===<br />
<br />
[[File:jsonhowto_fig4.png|thumb|right|400px|Figure 4: Example of dry run]]<br />
<br />
Click 'Import this' below the identifier that you would like to import. If you have completed the steps above a new panel will appear below the viewer. Confirm that this looks how you expect – '''this is what will be imported into the database'''.<br />
<br />
The final panel allows you to define which records will be imported. A type must be specified. If you are importing a file it may be necessary to do a separate import of each type. This is because different types have different fields attached to them, the drop down menus respond to the ones above them, presenting only the options available. If they are not completed in order it may result in unexpected behaviour.<br />
<br />
===Step 5.===<br />
When you are happy that you are importing the correct data into the correct field, click 'Submit'.<br />
You will be shown the results of your import. Click 'Import Json' to repeat the process for any other fields you wish to import.<br />
<br />
==Advanced Options==<br />
Ste_cd:The site code that imported items will have, is one is not specified in the data – uses the ARK default if not set. No site code: This check box will remove site codes from the import itemvalue, for use with chains.<br />
Regular Expression: This is used to extract the itemvalue from the ARK ID column – by default it grabs the first run of numbers in the field.<br />
Start at Arbitrary Number: If you wish to create a sequence of numbers starting from a given point, (including 1) you will need to specify this here, and the number to start on. A unique 'ARK ID' must still be specified in the data, but it will not be used.</div>Mikehttps://ark.lparchaeology.com/wiki/index.php?title=Importing_Text_Data&diff=3335Importing Text Data2015-09-14T11:34:45Z<p>Mike: /* Step 3. Determine ARK IDs */</p>
<hr />
<div>==ARK textfile importer user guide==<br />
Since v1.1.2 ARK ahs had a tool to import text based files. in either JSON or CSV <br />
Data represented in either json or csv format can be uploaded to the ARK database.The file can be filtered so that only certain records are uploaded. It can create new records or add to existing ones.<br />
<br />
===Step 1. Identify datafile===<br />
Specify the location of the file.The import page of ARK in 1.1.2 includes a function for importing data directly from a text file or a data stream available online. Currently the file must be available on a web server. (Fig. 1) Enter the location of the file, either a relative path from the ARK root eg data/uploads/your_data.csv or a remote web address eg http://www.example.com/ARK/data?format=json.<br />
Click Submit.<br />
<br />
Depend on the size of the file it may take some time to load into your browser. Once it is loaded it will be available to manipulate very quickly.A spinner will appear while this is happening.<br />
<br />
===Step 2. Determine 'Root' of the data structure.===<br />
<br />
[[File:jsonhowto_fig1.png|thumb|400px|right|Figure 1: File Picker]]<br />
<br />
The data you have will likely include information that relates to several ARK items. Within your data structure there will be a level that includes all of these item representations - in the same way as rows in a table. In a csv file where there is a single record per row the root will be 'root>items'. In a more complicated json file (for example one with file level metadata held in the root as well as the items) the records may be contained in an object within the root object. Json allows for extensible representation, so each 'row' in the json file may not include all the same fields. It may be necessary to use the filter tools to only import objects which have a certain criteria.<br />
<br />
The example shown here has many items in the items object in root. The first available row is used to identify each of the contained objects - in this case it is an ARK ID, but it could be anything and may not be unique if it is not unique in your structure. For example if field 1 in your table is a site code or context type.<br />
<br />
[[File:jsonhowto_fig2.png|thumb|400px|left|none|Figure 2: Objects in 'root>items']]<br />
<br />
In order to define this level you will need to navigate to it.The path at the top of the page describes where in the data structure you currently are.The large panel below the advanced options shows the identifiers held in the current level.At the root level of a CSV file the first column will be used as an identifier. Clicking on these will open the data attached to that identifier for viewing.(Fig. 2) It is possible to go up one level using the button at the end of the list of objects, or any part of the location at the top of the page can be used to navigate to that level.When the main window would contain more than 20 items it is truncated and only the first 20 are shown.<br />
<br />
Use the 'this is root' button in the navigation panel at the top of the page when the root is displayed in the main panel.<br />
<br />
===Step 3. Determine ARK IDs===<br />
<br />
[[File:jsonhowto_fig3.png|thumb|right|600px|Figure 3: Root and ARK ID specified]]<br />
<br />
Using the navigation method above find the path to the ARK id within your data. This will often be as simple as 'root>items>item1>Ark_id'.The object that you define the ARK ID in is not important. As long as the objects are structured consistently the ARK IDs will be retrieved automatically based on the root defined in the step above. so for instance the importer will import the ARK ID for METSUR_152 from root > items > METSUR_152 > ARK_id and then the ARK ID for METSUR_153 from root > items > METSUR_153 > ARK_id, and so on.<br />
<br />
<br />
If you do not have ARK IDs in your data refer to the advanced options.<br />
Otherwise, click 'This is ARK ID' button below the identifier that contains the ARK ID.<br />
<br />
===Step 4. Test Import===<br />
Click 'Import this' below the identifier that you would like to import. If you have completed the steps above a new panel will appear below the viewer. Confirm that this looks how you expect – this is what will be imported into the database.<br />
<br />
The final panel allows you to define which records will be imported.A type must be specified. If you are importing a file it may be necessary to do a separate import of each type. This is because different types have different fields attached to them, the drop down menus respond to the ones above them, presenting only the options available. If they are not completed in order it may result in unexpected behaviour.<br />
<br />
===Step 5.===<br />
When you are happy that you are importing the correct data into the correct field, click 'Submit'.<br />
You will be shown the results of your import. Click 'Import Json' to repeat the process for any other fields you wish to import.<br />
<br />
==Advanced Options==<br />
Ste_cd:The site code that imported items will have, is one is not specified in the data – uses the ARK default if not set. No site code: This check box will remove site codes from the import itemvalue, for use with chains.<br />
Regular Expression: This is used to extract the itemvalue from the ARK ID column – by default it grabs the first run of numbers in the field.<br />
Start at Arbitrary Number: If you wish to create a sequence of numbers starting from a given point, (including 1) you will need to specify this here, and the number to start on. A unique 'ARK ID' must still be specified in the data, but it will not be used.</div>Mikehttps://ark.lparchaeology.com/wiki/index.php?title=Importing_Text_Data&diff=3334Importing Text Data2015-09-14T11:34:18Z<p>Mike: /* Step 3. Determine ARK IDs */</p>
<hr />
<div>==ARK textfile importer user guide==<br />
Since v1.1.2 ARK ahs had a tool to import text based files. in either JSON or CSV <br />
Data represented in either json or csv format can be uploaded to the ARK database.The file can be filtered so that only certain records are uploaded. It can create new records or add to existing ones.<br />
<br />
===Step 1. Identify datafile===<br />
Specify the location of the file.The import page of ARK in 1.1.2 includes a function for importing data directly from a text file or a data stream available online. Currently the file must be available on a web server. (Fig. 1) Enter the location of the file, either a relative path from the ARK root eg data/uploads/your_data.csv or a remote web address eg http://www.example.com/ARK/data?format=json.<br />
Click Submit.<br />
<br />
Depend on the size of the file it may take some time to load into your browser. Once it is loaded it will be available to manipulate very quickly.A spinner will appear while this is happening.<br />
<br />
===Step 2. Determine 'Root' of the data structure.===<br />
<br />
[[File:jsonhowto_fig1.png|thumb|400px|right|Figure 1: File Picker]]<br />
<br />
The data you have will likely include information that relates to several ARK items. Within your data structure there will be a level that includes all of these item representations - in the same way as rows in a table. In a csv file where there is a single record per row the root will be 'root>items'. In a more complicated json file (for example one with file level metadata held in the root as well as the items) the records may be contained in an object within the root object. Json allows for extensible representation, so each 'row' in the json file may not include all the same fields. It may be necessary to use the filter tools to only import objects which have a certain criteria.<br />
<br />
The example shown here has many items in the items object in root. The first available row is used to identify each of the contained objects - in this case it is an ARK ID, but it could be anything and may not be unique if it is not unique in your structure. For example if field 1 in your table is a site code or context type.<br />
<br />
[[File:jsonhowto_fig2.png|thumb|400px|left|none|Figure 2: Objects in 'root>items']]<br />
<br />
In order to define this level you will need to navigate to it.The path at the top of the page describes where in the data structure you currently are.The large panel below the advanced options shows the identifiers held in the current level.At the root level of a CSV file the first column will be used as an identifier. Clicking on these will open the data attached to that identifier for viewing.(Fig. 2) It is possible to go up one level using the button at the end of the list of objects, or any part of the location at the top of the page can be used to navigate to that level.When the main window would contain more than 20 items it is truncated and only the first 20 are shown.<br />
<br />
Use the 'this is root' button in the navigation panel at the top of the page when the root is displayed in the main panel.<br />
<br />
===Step 3. Determine ARK IDs===<br />
Using the navigation method above find the path to the ARK id within your data. This will often be as simple as 'root>items>item1>Ark_id'.The object that you define the ARK ID in is not important. As long as the objects are structured consistently the ARK IDs will be retrieved automatically based on the root defined in the step above. so for instance the importer will import the ARK ID for METSUR_152 from root > items > METSUR_152 > ARK_id and then the ARK ID for METSUR_153 from root > items > METSUR_153 > ARK_id, and so on.<br />
<br />
[[File:jsonhowto_fig3.png|thumb|right|400px|Figure 3: Root and ARK ID specified]]<br />
<br />
If you do not have ARK IDs in your data refer to the advanced options.<br />
Otherwise, click 'This is ARK ID' button below the identifier that contains the ARK ID.<br />
<br />
===Step 4. Test Import===<br />
Click 'Import this' below the identifier that you would like to import. If you have completed the steps above a new panel will appear below the viewer. Confirm that this looks how you expect – this is what will be imported into the database.<br />
<br />
The final panel allows you to define which records will be imported.A type must be specified. If you are importing a file it may be necessary to do a separate import of each type. This is because different types have different fields attached to them, the drop down menus respond to the ones above them, presenting only the options available. If they are not completed in order it may result in unexpected behaviour.<br />
<br />
===Step 5.===<br />
When you are happy that you are importing the correct data into the correct field, click 'Submit'.<br />
You will be shown the results of your import. Click 'Import Json' to repeat the process for any other fields you wish to import.<br />
<br />
==Advanced Options==<br />
Ste_cd:The site code that imported items will have, is one is not specified in the data – uses the ARK default if not set. No site code: This check box will remove site codes from the import itemvalue, for use with chains.<br />
Regular Expression: This is used to extract the itemvalue from the ARK ID column – by default it grabs the first run of numbers in the field.<br />
Start at Arbitrary Number: If you wish to create a sequence of numbers starting from a given point, (including 1) you will need to specify this here, and the number to start on. A unique 'ARK ID' must still be specified in the data, but it will not be used.</div>Mikehttps://ark.lparchaeology.com/wiki/index.php?title=Importing_Text_Data&diff=3333Importing Text Data2015-09-14T11:32:10Z<p>Mike: /* Step 2. Determine 'Root' of the data structure. */</p>
<hr />
<div>==ARK textfile importer user guide==<br />
Since v1.1.2 ARK ahs had a tool to import text based files. in either JSON or CSV <br />
Data represented in either json or csv format can be uploaded to the ARK database.The file can be filtered so that only certain records are uploaded. It can create new records or add to existing ones.<br />
<br />
===Step 1. Identify datafile===<br />
Specify the location of the file.The import page of ARK in 1.1.2 includes a function for importing data directly from a text file or a data stream available online. Currently the file must be available on a web server. (Fig. 1) Enter the location of the file, either a relative path from the ARK root eg data/uploads/your_data.csv or a remote web address eg http://www.example.com/ARK/data?format=json.<br />
Click Submit.<br />
<br />
Depend on the size of the file it may take some time to load into your browser. Once it is loaded it will be available to manipulate very quickly.A spinner will appear while this is happening.<br />
<br />
===Step 2. Determine 'Root' of the data structure.===<br />
<br />
[[File:jsonhowto_fig1.png|thumb|400px|right|Figure 1: File Picker]]<br />
<br />
The data you have will likely include information that relates to several ARK items. Within your data structure there will be a level that includes all of these item representations - in the same way as rows in a table. In a csv file where there is a single record per row the root will be 'root>items'. In a more complicated json file (for example one with file level metadata held in the root as well as the items) the records may be contained in an object within the root object. Json allows for extensible representation, so each 'row' in the json file may not include all the same fields. It may be necessary to use the filter tools to only import objects which have a certain criteria.<br />
<br />
The example shown here has many items in the items object in root. The first available row is used to identify each of the contained objects - in this case it is an ARK ID, but it could be anything and may not be unique if it is not unique in your structure. For example if field 1 in your table is a site code or context type.<br />
<br />
[[File:jsonhowto_fig2.png|thumb|400px|left|none|Figure 2: Objects in 'root>items']]<br />
<br />
In order to define this level you will need to navigate to it.The path at the top of the page describes where in the data structure you currently are.The large panel below the advanced options shows the identifiers held in the current level.At the root level of a CSV file the first column will be used as an identifier. Clicking on these will open the data attached to that identifier for viewing.(Fig. 2) It is possible to go up one level using the button at the end of the list of objects, or any part of the location at the top of the page can be used to navigate to that level.When the main window would contain more than 20 items it is truncated and only the first 20 are shown.<br />
<br />
Use the 'this is root' button in the navigation panel at the top of the page when the root is displayed in the main panel.<br />
<br />
===Step 3. Determine ARK IDs===<br />
Using the navigation method above find the path to the ARK id within your data.This will often be as simple as 'root>items>item1>Ark_id'.The object that you define the ARK ID in is not important.As long as the objects are structured consistently the ARK IDs will be retrieved automatically based on the root defined in the step above.<br />
<br />
[[File:jsonhowto_fig3.png|thumb|right|850px|Figure 3: Root and ARK ID specified]]<br />
<br />
If you do not have ARK IDs in your data refer to the advanced options.<br />
Otherwise, click 'This is ARK ID' button below the identifier that contains the ARK ID.<br />
<br />
===Step 4. Test Import===<br />
Click 'Import this' below the identifier that you would like to import. If you have completed the steps above a new panel will appear below the viewer. Confirm that this looks how you expect – this is what will be imported into the database.<br />
<br />
The final panel allows you to define which records will be imported.A type must be specified. If you are importing a file it may be necessary to do a separate import of each type. This is because different types have different fields attached to them, the drop down menus respond to the ones above them, presenting only the options available. If they are not completed in order it may result in unexpected behaviour.<br />
<br />
===Step 5.===<br />
When you are happy that you are importing the correct data into the correct field, click 'Submit'.<br />
You will be shown the results of your import. Click 'Import Json' to repeat the process for any other fields you wish to import.<br />
<br />
==Advanced Options==<br />
Ste_cd:The site code that imported items will have, is one is not specified in the data – uses the ARK default if not set. No site code: This check box will remove site codes from the import itemvalue, for use with chains.<br />
Regular Expression: This is used to extract the itemvalue from the ARK ID column – by default it grabs the first run of numbers in the field.<br />
Start at Arbitrary Number: If you wish to create a sequence of numbers starting from a given point, (including 1) you will need to specify this here, and the number to start on. A unique 'ARK ID' must still be specified in the data, but it will not be used.</div>Mikehttps://ark.lparchaeology.com/wiki/index.php?title=Importing_Text_Data&diff=3332Importing Text Data2015-09-14T11:24:57Z<p>Mike: /* Step 2. Determine 'Root' of the data structure. */</p>
<hr />
<div>==ARK textfile importer user guide==<br />
Since v1.1.2 ARK ahs had a tool to import text based files. in either JSON or CSV <br />
Data represented in either json or csv format can be uploaded to the ARK database.The file can be filtered so that only certain records are uploaded. It can create new records or add to existing ones.<br />
<br />
===Step 1. Identify datafile===<br />
Specify the location of the file.The import page of ARK in 1.1.2 includes a function for importing data directly from a text file or a data stream available online. Currently the file must be available on a web server. (Fig. 1) Enter the location of the file, either a relative path from the ARK root eg data/uploads/your_data.csv or a remote web address eg http://www.example.com/ARK/data?format=json.<br />
Click Submit.<br />
<br />
Depend on the size of the file it may take some time to load into your browser. Once it is loaded it will be available to manipulate very quickly.A spinner will appear while this is happening.<br />
<br />
===Step 2. Determine 'Root' of the data structure.===<br />
<br />
[[File:jsonhowto_fig1.png|thumb|400px|right|Figure 1: File Picker]]<br />
<br />
The data you have will likely include data that relates to several ARK items.The level that these repeat in the data structure must be defined so that ARK can import all the records automatically. In a csv file where there is a single record per row the root will be 'root>items'. In a more complicated json file (for example one with file level metadata held in the root as well as the items) the records may be contained in a object within the root object.<br />
<br />
[[File:jsonhowto_fig2.png|thumb|400px|left|none|Figure 2: Objects in 'root>items']]<br />
<br />
In order to define this level you will need to navigate to it.The path at the top of the page describes where in the data structure you currently are.The large panel below the advanced options shows the identifiers held in the current level.At the root level of a CSV file the first column will be used as an identifier. Clicking on these will open the data attached to that identifier for viewing.(Fig. 2) It is possible to go up one level using the button at the end of the list of objects, or any part of the location at the top of the page can be used to navigate to that level.When the main window would contain more than 20 items it is truncated and only the first 20 are shown.<br />
<br />
Use the 'this is root' button in the navigation panel at the top of the page when the root is displayed in the main panel.<br />
<br />
===Step 3. Determine ARK IDs===<br />
Using the navigation method above find the path to the ARK id within your data.This will often be as simple as 'root>items>item1>Ark_id'.The object that you define the ARK ID in is not important.As long as the objects are structured consistently the ARK IDs will be retrieved automatically based on the root defined in the step above.<br />
<br />
[[File:jsonhowto_fig3.png|thumb|right|850px|Figure 3: Root and ARK ID specified]]<br />
<br />
If you do not have ARK IDs in your data refer to the advanced options.<br />
Otherwise, click 'This is ARK ID' button below the identifier that contains the ARK ID.<br />
<br />
===Step 4. Test Import===<br />
Click 'Import this' below the identifier that you would like to import. If you have completed the steps above a new panel will appear below the viewer. Confirm that this looks how you expect – this is what will be imported into the database.<br />
<br />
The final panel allows you to define which records will be imported.A type must be specified. If you are importing a file it may be necessary to do a separate import of each type. This is because different types have different fields attached to them, the drop down menus respond to the ones above them, presenting only the options available. If they are not completed in order it may result in unexpected behaviour.<br />
<br />
===Step 5.===<br />
When you are happy that you are importing the correct data into the correct field, click 'Submit'.<br />
You will be shown the results of your import. Click 'Import Json' to repeat the process for any other fields you wish to import.<br />
<br />
==Advanced Options==<br />
Ste_cd:The site code that imported items will have, is one is not specified in the data – uses the ARK default if not set. No site code: This check box will remove site codes from the import itemvalue, for use with chains.<br />
Regular Expression: This is used to extract the itemvalue from the ARK ID column – by default it grabs the first run of numbers in the field.<br />
Start at Arbitrary Number: If you wish to create a sequence of numbers starting from a given point, (including 1) you will need to specify this here, and the number to start on. A unique 'ARK ID' must still be specified in the data, but it will not be used.</div>Mikehttps://ark.lparchaeology.com/wiki/index.php?title=Importing_Text_Data&diff=3331Importing Text Data2015-09-14T11:24:17Z<p>Mike: /* Step 2. Determine 'Root' of the data structure. */</p>
<hr />
<div>==ARK textfile importer user guide==<br />
Since v1.1.2 ARK ahs had a tool to import text based files. in either JSON or CSV <br />
Data represented in either json or csv format can be uploaded to the ARK database.The file can be filtered so that only certain records are uploaded. It can create new records or add to existing ones.<br />
<br />
===Step 1. Identify datafile===<br />
Specify the location of the file.The import page of ARK in 1.1.2 includes a function for importing data directly from a text file or a data stream available online. Currently the file must be available on a web server. (Fig. 1) Enter the location of the file, either a relative path from the ARK root eg data/uploads/your_data.csv or a remote web address eg http://www.example.com/ARK/data?format=json.<br />
Click Submit.<br />
<br />
Depend on the size of the file it may take some time to load into your browser. Once it is loaded it will be available to manipulate very quickly.A spinner will appear while this is happening.<br />
<br />
===Step 2. Determine 'Root' of the data structure.===<br />
The data you have will likely include data that relates to several ARK items.The level that these repeat in the data structure must be defined so that ARK can import all the records automatically. In a csv file where there is a single record per row the root will be 'root>items'. In a more complicated json file (for example one with file level metadata held in the root as well as the items) the records may be contained in a object within the root object.<br />
<br />
[[File:jsonhowto_fig1.png|thumb|400px|right|Figure 1: File Picker]]<br />
<br />
In order to define this level you will need to navigate to it.The path at the top of the page describes where in the data structure you currently are.The large panel below the advanced options shows the identifiers held in the current level.At the root level of a CSV file the first column will be used as an identifier. Clicking on these will open the data attached to that identifier for viewing.(Fig. 2) It is possible to go up one level using the button at the end of the list of objects, or any part of the location at the top of the page can be used to navigate to that level.When the main window would contain more than 20 items it is truncated and only the first 20 are shown.<br />
<br />
[[File:jsonhowto_fig2.png|thumb|400px|right|none|Figure 2: Objects in 'root>items']]<br />
<br />
Use the 'this is root' button in the navigation panel at the top of the page when the root is displayed in the main panel.<br />
<br />
===Step 3. Determine ARK IDs===<br />
Using the navigation method above find the path to the ARK id within your data.This will often be as simple as 'root>items>item1>Ark_id'.The object that you define the ARK ID in is not important.As long as the objects are structured consistently the ARK IDs will be retrieved automatically based on the root defined in the step above.<br />
<br />
[[File:jsonhowto_fig3.png|thumb|right|850px|Figure 3: Root and ARK ID specified]]<br />
<br />
If you do not have ARK IDs in your data refer to the advanced options.<br />
Otherwise, click 'This is ARK ID' button below the identifier that contains the ARK ID.<br />
<br />
===Step 4. Test Import===<br />
Click 'Import this' below the identifier that you would like to import. If you have completed the steps above a new panel will appear below the viewer. Confirm that this looks how you expect – this is what will be imported into the database.<br />
<br />
The final panel allows you to define which records will be imported.A type must be specified. If you are importing a file it may be necessary to do a separate import of each type. This is because different types have different fields attached to them, the drop down menus respond to the ones above them, presenting only the options available. If they are not completed in order it may result in unexpected behaviour.<br />
<br />
===Step 5.===<br />
When you are happy that you are importing the correct data into the correct field, click 'Submit'.<br />
You will be shown the results of your import. Click 'Import Json' to repeat the process for any other fields you wish to import.<br />
<br />
==Advanced Options==<br />
Ste_cd:The site code that imported items will have, is one is not specified in the data – uses the ARK default if not set. No site code: This check box will remove site codes from the import itemvalue, for use with chains.<br />
Regular Expression: This is used to extract the itemvalue from the ARK ID column – by default it grabs the first run of numbers in the field.<br />
Start at Arbitrary Number: If you wish to create a sequence of numbers starting from a given point, (including 1) you will need to specify this here, and the number to start on. A unique 'ARK ID' must still be specified in the data, but it will not be used.</div>Mikehttps://ark.lparchaeology.com/wiki/index.php?title=Importing_Text_Data&diff=3330Importing Text Data2015-09-14T11:23:55Z<p>Mike: /* Step 3. Determine ARK IDs */</p>
<hr />
<div>==ARK textfile importer user guide==<br />
Since v1.1.2 ARK ahs had a tool to import text based files. in either JSON or CSV <br />
Data represented in either json or csv format can be uploaded to the ARK database.The file can be filtered so that only certain records are uploaded. It can create new records or add to existing ones.<br />
<br />
===Step 1. Identify datafile===<br />
Specify the location of the file.The import page of ARK in 1.1.2 includes a function for importing data directly from a text file or a data stream available online. Currently the file must be available on a web server. (Fig. 1) Enter the location of the file, either a relative path from the ARK root eg data/uploads/your_data.csv or a remote web address eg http://www.example.com/ARK/data?format=json.<br />
Click Submit.<br />
<br />
Depend on the size of the file it may take some time to load into your browser. Once it is loaded it will be available to manipulate very quickly.A spinner will appear while this is happening.<br />
<br />
===Step 2. Determine 'Root' of the data structure.===<br />
The data you have will likely include data that relates to several ARK items.The level that these repeat in the data structure must be defined so that ARK can import all the records automatically. In a csv file where there is a single record per row the root will be 'root>items'. In a more complicated json file (for example one with file level metadata held in the root as well as the items) the records may be contained in a object within the root object.<br />
<br />
[[File:jsonhowto_fig1.png|thumb|850px|right|Figure 1: File Picker]]<br />
<br />
In order to define this level you will need to navigate to it.The path at the top of the page describes where in the data structure you currently are.The large panel below the advanced options shows the identifiers held in the current level.At the root level of a CSV file the first column will be used as an identifier. Clicking on these will open the data attached to that identifier for viewing.(Fig. 2) It is possible to go up one level using the button at the end of the list of objects, or any part of the location at the top of the page can be used to navigate to that level.When the main window would contain more than 20 items it is truncated and only the first 20 are shown.<br />
<br />
[[File:jsonhowto_fig2.png|thumb|850px|right|none|Figure 2: Objects in 'root>items']]<br />
<br />
Use the 'this is root' button in the navigation panel at the top of the page when the root is displayed in the main panel.<br />
<br />
===Step 3. Determine ARK IDs===<br />
Using the navigation method above find the path to the ARK id within your data.This will often be as simple as 'root>items>item1>Ark_id'.The object that you define the ARK ID in is not important.As long as the objects are structured consistently the ARK IDs will be retrieved automatically based on the root defined in the step above.<br />
<br />
[[File:jsonhowto_fig3.png|thumb|right|850px|Figure 3: Root and ARK ID specified]]<br />
<br />
If you do not have ARK IDs in your data refer to the advanced options.<br />
Otherwise, click 'This is ARK ID' button below the identifier that contains the ARK ID.<br />
<br />
===Step 4. Test Import===<br />
Click 'Import this' below the identifier that you would like to import. If you have completed the steps above a new panel will appear below the viewer. Confirm that this looks how you expect – this is what will be imported into the database.<br />
<br />
The final panel allows you to define which records will be imported.A type must be specified. If you are importing a file it may be necessary to do a separate import of each type. This is because different types have different fields attached to them, the drop down menus respond to the ones above them, presenting only the options available. If they are not completed in order it may result in unexpected behaviour.<br />
<br />
===Step 5.===<br />
When you are happy that you are importing the correct data into the correct field, click 'Submit'.<br />
You will be shown the results of your import. Click 'Import Json' to repeat the process for any other fields you wish to import.<br />
<br />
==Advanced Options==<br />
Ste_cd:The site code that imported items will have, is one is not specified in the data – uses the ARK default if not set. No site code: This check box will remove site codes from the import itemvalue, for use with chains.<br />
Regular Expression: This is used to extract the itemvalue from the ARK ID column – by default it grabs the first run of numbers in the field.<br />
Start at Arbitrary Number: If you wish to create a sequence of numbers starting from a given point, (including 1) you will need to specify this here, and the number to start on. A unique 'ARK ID' must still be specified in the data, but it will not be used.</div>Mikehttps://ark.lparchaeology.com/wiki/index.php?title=Importing_Text_Data&diff=3329Importing Text Data2015-09-14T11:23:02Z<p>Mike: /* Step 2. Determine 'Root' of the data structure. */</p>
<hr />
<div>==ARK textfile importer user guide==<br />
Since v1.1.2 ARK ahs had a tool to import text based files. in either JSON or CSV <br />
Data represented in either json or csv format can be uploaded to the ARK database.The file can be filtered so that only certain records are uploaded. It can create new records or add to existing ones.<br />
<br />
===Step 1. Identify datafile===<br />
Specify the location of the file.The import page of ARK in 1.1.2 includes a function for importing data directly from a text file or a data stream available online. Currently the file must be available on a web server. (Fig. 1) Enter the location of the file, either a relative path from the ARK root eg data/uploads/your_data.csv or a remote web address eg http://www.example.com/ARK/data?format=json.<br />
Click Submit.<br />
<br />
Depend on the size of the file it may take some time to load into your browser. Once it is loaded it will be available to manipulate very quickly.A spinner will appear while this is happening.<br />
<br />
===Step 2. Determine 'Root' of the data structure.===<br />
The data you have will likely include data that relates to several ARK items.The level that these repeat in the data structure must be defined so that ARK can import all the records automatically. In a csv file where there is a single record per row the root will be 'root>items'. In a more complicated json file (for example one with file level metadata held in the root as well as the items) the records may be contained in a object within the root object.<br />
<br />
[[File:jsonhowto_fig1.png|thumb|850px|right|Figure 1: File Picker]]<br />
<br />
In order to define this level you will need to navigate to it.The path at the top of the page describes where in the data structure you currently are.The large panel below the advanced options shows the identifiers held in the current level.At the root level of a CSV file the first column will be used as an identifier. Clicking on these will open the data attached to that identifier for viewing.(Fig. 2) It is possible to go up one level using the button at the end of the list of objects, or any part of the location at the top of the page can be used to navigate to that level.When the main window would contain more than 20 items it is truncated and only the first 20 are shown.<br />
<br />
[[File:jsonhowto_fig2.png|thumb|850px|right|none|Figure 2: Objects in 'root>items']]<br />
<br />
Use the 'this is root' button in the navigation panel at the top of the page when the root is displayed in the main panel.<br />
<br />
===Step 3. Determine ARK IDs===<br />
Using the navigation method above find the path to the ARK id within your data.This will often be as simple as 'root>items>item1>Ark_id'.The object that you define the ARK ID in is not important.As long as the objects are structured consistently the ARK IDs will be retrieved automatically based on the root defined in the step above.<br />
<br />
Figure 3: Root and ARK ID specified<br />
If you do not have ARK IDs in your data refer to the advanced options.<br />
Otherwise, click 'This is ARK ID' button below the identifier that contains the ARK ID.<br />
<br />
===Step 4. Test Import===<br />
Click 'Import this' below the identifier that you would like to import. If you have completed the steps above a new panel will appear below the viewer. Confirm that this looks how you expect – this is what will be imported into the database.<br />
<br />
The final panel allows you to define which records will be imported.A type must be specified. If you are importing a file it may be necessary to do a separate import of each type. This is because different types have different fields attached to them, the drop down menus respond to the ones above them, presenting only the options available. If they are not completed in order it may result in unexpected behaviour.<br />
<br />
===Step 5.===<br />
When you are happy that you are importing the correct data into the correct field, click 'Submit'.<br />
You will be shown the results of your import. Click 'Import Json' to repeat the process for any other fields you wish to import.<br />
<br />
==Advanced Options==<br />
Ste_cd:The site code that imported items will have, is one is not specified in the data – uses the ARK default if not set. No site code: This check box will remove site codes from the import itemvalue, for use with chains.<br />
Regular Expression: This is used to extract the itemvalue from the ARK ID column – by default it grabs the first run of numbers in the field.<br />
Start at Arbitrary Number: If you wish to create a sequence of numbers starting from a given point, (including 1) you will need to specify this here, and the number to start on. A unique 'ARK ID' must still be specified in the data, but it will not be used.</div>Mikehttps://ark.lparchaeology.com/wiki/index.php?title=Importing_Text_Data&diff=3328Importing Text Data2015-09-14T11:14:36Z<p>Mike: /* Step 2. Determine 'Root' of the data structure. */</p>
<hr />
<div>==ARK textfile importer user guide==<br />
Since v1.1.2 ARK ahs had a tool to import text based files. in either JSON or CSV <br />
Data represented in either json or csv format can be uploaded to the ARK database.The file can be filtered so that only certain records are uploaded. It can create new records or add to existing ones.<br />
<br />
===Step 1. Identify datafile===<br />
Specify the location of the file.The import page of ARK in 1.1.2 includes a function for importing data directly from a text file or a data stream available online. Currently the file must be available on a web server. (Fig. 1) Enter the location of the file, either a relative path from the ARK root eg data/uploads/your_data.csv or a remote web address eg http://www.example.com/ARK/data?format=json.<br />
Click Submit.<br />
<br />
Depend on the size of the file it may take some time to load into your browser. Once it is loaded it will be available to manipulate very quickly.A spinner will appear while this is happening.<br />
<br />
===Step 2. Determine 'Root' of the data structure.===<br />
The data you have will likely include data that relates to several ARK items.The level that these repeat in the data structure must be defined so that ARK can import all the records automatically. In a csv file where there is a single record per row the root will be 'root>items'. In a more complicated json file (for example one with file level metadata held in the root as well as the items) the records may be contained in a object within the root object.<br />
<br />
[[File:jsonhowto_fig1.png|thumb|left|Figure 1: File Picker]]<br />
<br />
In order to define this level you will need to navigate to it.The path at the top of the page describes where in the data structure you currently are.The large panel below the advanced options shows the identifiers held in the current level.At the root level of a CSV file the first column will be used as an identifier. Clicking on these will open the data attached to that identifier for viewing.(Fig. 2) It is possible to go up one level using the button at the end of the list of objects, or any part of the location at the top of the page can be used to navigate to that level.When the main window would contain more than 20 items it is truncated and only the first 20 are shown.<br />
<br />
[[File:jsonhowto_fig2.png|thumb|right|none|Figure 2: Objects in 'root>items']]<br />
<br />
Use the 'this is root' button in the navigation panel at the top of the page when the root is displayed in the main panel.<br />
<br />
===Step 3. Determine ARK IDs===<br />
Using the navigation method above find the path to the ARK id within your data.This will often be as simple as 'root>items>item1>Ark_id'.The object that you define the ARK ID in is not important.As long as the objects are structured consistently the ARK IDs will be retrieved automatically based on the root defined in the step above.<br />
<br />
Figure 3: Root and ARK ID specified<br />
If you do not have ARK IDs in your data refer to the advanced options.<br />
Otherwise, click 'This is ARK ID' button below the identifier that contains the ARK ID.<br />
<br />
===Step 4. Test Import===<br />
Click 'Import this' below the identifier that you would like to import. If you have completed the steps above a new panel will appear below the viewer. Confirm that this looks how you expect – this is what will be imported into the database.<br />
<br />
The final panel allows you to define which records will be imported.A type must be specified. If you are importing a file it may be necessary to do a separate import of each type. This is because different types have different fields attached to them, the drop down menus respond to the ones above them, presenting only the options available. If they are not completed in order it may result in unexpected behaviour.<br />
<br />
===Step 5.===<br />
When you are happy that you are importing the correct data into the correct field, click 'Submit'.<br />
You will be shown the results of your import. Click 'Import Json' to repeat the process for any other fields you wish to import.<br />
<br />
==Advanced Options==<br />
Ste_cd:The site code that imported items will have, is one is not specified in the data – uses the ARK default if not set. No site code: This check box will remove site codes from the import itemvalue, for use with chains.<br />
Regular Expression: This is used to extract the itemvalue from the ARK ID column – by default it grabs the first run of numbers in the field.<br />
Start at Arbitrary Number: If you wish to create a sequence of numbers starting from a given point, (including 1) you will need to specify this here, and the number to start on. A unique 'ARK ID' must still be specified in the data, but it will not be used.</div>Mikehttps://ark.lparchaeology.com/wiki/index.php?title=Importing_Text_Data&diff=3327Importing Text Data2015-09-14T11:13:51Z<p>Mike: /* Step 2. Determine 'Root' of the data structure. */</p>
<hr />
<div>==ARK textfile importer user guide==<br />
Since v1.1.2 ARK ahs had a tool to import text based files. in either JSON or CSV <br />
Data represented in either json or csv format can be uploaded to the ARK database.The file can be filtered so that only certain records are uploaded. It can create new records or add to existing ones.<br />
<br />
===Step 1. Identify datafile===<br />
Specify the location of the file.The import page of ARK in 1.1.2 includes a function for importing data directly from a text file or a data stream available online. Currently the file must be available on a web server. (Fig. 1) Enter the location of the file, either a relative path from the ARK root eg data/uploads/your_data.csv or a remote web address eg http://www.example.com/ARK/data?format=json.<br />
Click Submit.<br />
<br />
Depend on the size of the file it may take some time to load into your browser. Once it is loaded it will be available to manipulate very quickly.A spinner will appear while this is happening.<br />
<br />
===Step 2. Determine 'Root' of the data structure.===<br />
The data you have will likely include data that relates to several ARK items.The level that these repeat in the data structure must be defined so that ARK can import all the records automatically. In a csv file where there is a single record per row the root will be 'root>items'. In a more complicated json file (for example one with file level metadata held in the root as well as the items) the records may be contained in a object within the root object.<br />
<br />
[[File:jsonhowto_fig1.png|thumb|left|Figure 1: File Picker]]<br />
<br />
In order to define this level you will need to navigate to it.The path at the top of the page describes where in the data structure you currently are.The large panel below the advanced options shows the identifiers held in the current level.At the root level of a CSV file the first column will be used as an identifier. Clicking on these will open the data attached to that identifier for viewing.(Fig. 2) It is possible to go up one level using the button at the end of the list of objects, or any part of the location at the top of the page can be used to navigate to that level.When the main window would contain more than 20 items it is truncated and only the first 20 are shown.<br />
<br />
[[File:jsonhowto_fig2.png|thumb|none|Figure 2: Objects in 'root>items']]<br />
<br />
Use the 'this is root' button in the navigation panel at the top of the page when the root is displayed in the main panel.<br />
<br />
===Step 3. Determine ARK IDs===<br />
Using the navigation method above find the path to the ARK id within your data.This will often be as simple as 'root>items>item1>Ark_id'.The object that you define the ARK ID in is not important.As long as the objects are structured consistently the ARK IDs will be retrieved automatically based on the root defined in the step above.<br />
<br />
Figure 3: Root and ARK ID specified<br />
If you do not have ARK IDs in your data refer to the advanced options.<br />
Otherwise, click 'This is ARK ID' button below the identifier that contains the ARK ID.<br />
<br />
===Step 4. Test Import===<br />
Click 'Import this' below the identifier that you would like to import. If you have completed the steps above a new panel will appear below the viewer. Confirm that this looks how you expect – this is what will be imported into the database.<br />
<br />
The final panel allows you to define which records will be imported.A type must be specified. If you are importing a file it may be necessary to do a separate import of each type. This is because different types have different fields attached to them, the drop down menus respond to the ones above them, presenting only the options available. If they are not completed in order it may result in unexpected behaviour.<br />
<br />
===Step 5.===<br />
When you are happy that you are importing the correct data into the correct field, click 'Submit'.<br />
You will be shown the results of your import. Click 'Import Json' to repeat the process for any other fields you wish to import.<br />
<br />
==Advanced Options==<br />
Ste_cd:The site code that imported items will have, is one is not specified in the data – uses the ARK default if not set. No site code: This check box will remove site codes from the import itemvalue, for use with chains.<br />
Regular Expression: This is used to extract the itemvalue from the ARK ID column – by default it grabs the first run of numbers in the field.<br />
Start at Arbitrary Number: If you wish to create a sequence of numbers starting from a given point, (including 1) you will need to specify this here, and the number to start on. A unique 'ARK ID' must still be specified in the data, but it will not be used.</div>Mikehttps://ark.lparchaeology.com/wiki/index.php?title=Importing_Text_Data&diff=3326Importing Text Data2015-09-14T11:13:23Z<p>Mike: /* Step 2. Determine 'Root' of the data structure. */</p>
<hr />
<div>==ARK textfile importer user guide==<br />
Since v1.1.2 ARK ahs had a tool to import text based files. in either JSON or CSV <br />
Data represented in either json or csv format can be uploaded to the ARK database.The file can be filtered so that only certain records are uploaded. It can create new records or add to existing ones.<br />
<br />
===Step 1. Identify datafile===<br />
Specify the location of the file.The import page of ARK in 1.1.2 includes a function for importing data directly from a text file or a data stream available online. Currently the file must be available on a web server. (Fig. 1) Enter the location of the file, either a relative path from the ARK root eg data/uploads/your_data.csv or a remote web address eg http://www.example.com/ARK/data?format=json.<br />
Click Submit.<br />
<br />
Depend on the size of the file it may take some time to load into your browser. Once it is loaded it will be available to manipulate very quickly.A spinner will appear while this is happening.<br />
<br />
===Step 2. Determine 'Root' of the data structure.===<br />
The data you have will likely include data that relates to several ARK items.The level that these repeat in the data structure must be defined so that ARK can import all the records automatically. In a csv file where there is a single record per row the root will be 'root>items'. In a more complicated json file (for example one with file level metadata held in the root as well as the items) the records may be contained in a object within the root object.<br />
<br />
[[File:jsonhowto_fig1.png|thumb|left|Figure 1: File Picker]]<br />
<br />
In order to define this level you will need to navigate to it.The path at the top of the page describes where in the data structure you currently are.The large panel below the advanced options shows the identifiers held in the current level.At the root level of a CSV file the first column will be used as an identifier. Clicking on these will open the data attached to that identifier for viewing.(Fig. 2) It is possible to go up one level using the button at the end of the list of objects, or any part of the location at the top of the page can be used to navigate to that level.When the main window would contain more than 20 items it is truncated and only the first 20 are shown.<br />
<br />
[[File:jsonhowto_fig2.png|thumb|Figure 2: Objects in 'root>items']]<br />
<br />
Use the 'this is root' button in the navigation panel at the top of the page when the root is displayed in the main panel.<br />
<br />
===Step 3. Determine ARK IDs===<br />
Using the navigation method above find the path to the ARK id within your data.This will often be as simple as 'root>items>item1>Ark_id'.The object that you define the ARK ID in is not important.As long as the objects are structured consistently the ARK IDs will be retrieved automatically based on the root defined in the step above.<br />
<br />
Figure 3: Root and ARK ID specified<br />
If you do not have ARK IDs in your data refer to the advanced options.<br />
Otherwise, click 'This is ARK ID' button below the identifier that contains the ARK ID.<br />
<br />
===Step 4. Test Import===<br />
Click 'Import this' below the identifier that you would like to import. If you have completed the steps above a new panel will appear below the viewer. Confirm that this looks how you expect – this is what will be imported into the database.<br />
<br />
The final panel allows you to define which records will be imported.A type must be specified. If you are importing a file it may be necessary to do a separate import of each type. This is because different types have different fields attached to them, the drop down menus respond to the ones above them, presenting only the options available. If they are not completed in order it may result in unexpected behaviour.<br />
<br />
===Step 5.===<br />
When you are happy that you are importing the correct data into the correct field, click 'Submit'.<br />
You will be shown the results of your import. Click 'Import Json' to repeat the process for any other fields you wish to import.<br />
<br />
==Advanced Options==<br />
Ste_cd:The site code that imported items will have, is one is not specified in the data – uses the ARK default if not set. No site code: This check box will remove site codes from the import itemvalue, for use with chains.<br />
Regular Expression: This is used to extract the itemvalue from the ARK ID column – by default it grabs the first run of numbers in the field.<br />
Start at Arbitrary Number: If you wish to create a sequence of numbers starting from a given point, (including 1) you will need to specify this here, and the number to start on. A unique 'ARK ID' must still be specified in the data, but it will not be used.</div>Mikehttps://ark.lparchaeology.com/wiki/index.php?title=Importing_Text_Data&diff=3325Importing Text Data2015-09-14T11:13:02Z<p>Mike: /* Step 2. Determine 'Root' of the data structure. */</p>
<hr />
<div>==ARK textfile importer user guide==<br />
Since v1.1.2 ARK ahs had a tool to import text based files. in either JSON or CSV <br />
Data represented in either json or csv format can be uploaded to the ARK database.The file can be filtered so that only certain records are uploaded. It can create new records or add to existing ones.<br />
<br />
===Step 1. Identify datafile===<br />
Specify the location of the file.The import page of ARK in 1.1.2 includes a function for importing data directly from a text file or a data stream available online. Currently the file must be available on a web server. (Fig. 1) Enter the location of the file, either a relative path from the ARK root eg data/uploads/your_data.csv or a remote web address eg http://www.example.com/ARK/data?format=json.<br />
Click Submit.<br />
<br />
Depend on the size of the file it may take some time to load into your browser. Once it is loaded it will be available to manipulate very quickly.A spinner will appear while this is happening.<br />
<br />
===Step 2. Determine 'Root' of the data structure.===<br />
The data you have will likely include data that relates to several ARK items.The level that these repeat in the data structure must be defined so that ARK can import all the records automatically. In a csv file where there is a single record per row the root will be 'root>items'. In a more complicated json file (for example one with file level metadata held in the root as well as the items) the records may be contained in a object within the root object.<br />
<br />
[[File:jsonhowto_fig1.png|thumb|left|Figure 1: File Picker]]<br />
<br />
In order to define this level you will need to navigate to it.The path at the top of the page describes where in the data structure you currently are.The large panel below the advanced options shows the identifiers held in the current level.At the root level of a CSV file the first column will be used as an identifier. Clicking on these will open the data attached to that identifier for viewing.(Fig. 2) It is possible to go up one level using the button at the end of the list of objects, or any part of the location at the top of the page can be used to navigate to that level.When the main window would contain more than 20 items it is truncated and only the first 20 are shown.<br />
<br />
Use the 'this is root' button in the navigation panel at the top of the page when the root is displayed in the main panel.<br />
<br />
[[File:jsonhowto_fig2.png|thumb|Figure 2: Objects in 'root>items']]<br />
<br />
===Step 3. Determine ARK IDs===<br />
Using the navigation method above find the path to the ARK id within your data.This will often be as simple as 'root>items>item1>Ark_id'.The object that you define the ARK ID in is not important.As long as the objects are structured consistently the ARK IDs will be retrieved automatically based on the root defined in the step above.<br />
<br />
Figure 3: Root and ARK ID specified<br />
If you do not have ARK IDs in your data refer to the advanced options.<br />
Otherwise, click 'This is ARK ID' button below the identifier that contains the ARK ID.<br />
<br />
===Step 4. Test Import===<br />
Click 'Import this' below the identifier that you would like to import. If you have completed the steps above a new panel will appear below the viewer. Confirm that this looks how you expect – this is what will be imported into the database.<br />
<br />
The final panel allows you to define which records will be imported.A type must be specified. If you are importing a file it may be necessary to do a separate import of each type. This is because different types have different fields attached to them, the drop down menus respond to the ones above them, presenting only the options available. If they are not completed in order it may result in unexpected behaviour.<br />
<br />
===Step 5.===<br />
When you are happy that you are importing the correct data into the correct field, click 'Submit'.<br />
You will be shown the results of your import. Click 'Import Json' to repeat the process for any other fields you wish to import.<br />
<br />
==Advanced Options==<br />
Ste_cd:The site code that imported items will have, is one is not specified in the data – uses the ARK default if not set. No site code: This check box will remove site codes from the import itemvalue, for use with chains.<br />
Regular Expression: This is used to extract the itemvalue from the ARK ID column – by default it grabs the first run of numbers in the field.<br />
Start at Arbitrary Number: If you wish to create a sequence of numbers starting from a given point, (including 1) you will need to specify this here, and the number to start on. A unique 'ARK ID' must still be specified in the data, but it will not be used.</div>Mike