Category Archives: Support

In this category I will try to keep track of problems I ran into (missing dependencies / memory tweaks / os-specific configuration) when working with tools described on this site.

Thin results – random sample (lines) from text file

When working with corpora it is sometimes useful to be able to generate random samples from corpus results for manual analysis (e.g. to determine distribution percentages or recall/precision of queries). BNCweb, CQPweb or (No)SketchEngine provide a thin function for this purpose. However, if the results of corpus queries are only available as text files, there is a random thinning option available as part of GNU coreutils. The examples below create a random sample of 100 lines (adapt sample size according to your project’s needs). The reliability of manually checked results can be improved by obtaining several samples of 100 lines (typically 2-3) and using averaged scores.

On Linux, there is a very easy straight-forward way to achieve this (type: man shuf for details):
cd path_to_text_file
shuf -n 100 results.txt

In order to save the random sample into a new text file, specify an output file:
shuf -n 100 -o random_sample.txt results.txt

On Mac OSX, it is slightly more complicated, as a Linux-like package manager (e.g. Homebrew) and the coreutils package have to be installed first (gshuf Tutorial OSX and corresponding random_sample.zip for novice users who are not familiar with OSX terminal). Once the gshuf command is available, the invocation is anologous (type: man gshuf for details):
cd path_to_text_file
gshuf -n 100 results.txt

In order to save the random sample into a new text file, specify an output file:
gshuf -n 100 -o random_sample.txt results.txt

On Windows, the following Python code snipped could be used to achieve a similar result (please let me know if there are any built-in options):

Random sampler (Python, Algorithm 1)

Random sampler (Python, Algorithm 1)

Source: http://metadatascience.com/2014/02/27/random-sampling-from-very-large-files [accessed: 31/05/2015]

CQPweb tutorial (German)

Noah Bubenhofer's CQPweb Tutorial (German)

Noah Bubenhofer’s CQPweb Tutorial (German)

Linkt to Noah Bubenhofer’s CQPweb Tutorial (German)

CQPweb

2023-07-23 Developer / Project Head: Andrew Hardie
Purpose/Version/Date web interface to cwb stable: 3.2.43 dev: 3.3.15 [r1848] 31 Dec 2021 / 23 July 2023 Platform/License web-based (also on localhost) open source License: GNU GPLv2+ Price/Availability free Programming Language(s): php, mysql, Perl Key features: SERVER INSTALLATION, MANAGE YOUR OWN CORPORA, WEB INTERFACE, CQP QUERIES Website: CQPweb project page Website: CQPWeb SVN Repository Website: UCREL Lancaster Corpus Server (free access to a lot of corpus resources after registration, including the extended Brown-family of corpora) Website: CQPweb at Beijing Foreign Studies University – Large Number of publicly accessible corpora (username: test, password: test) Website: CQPweb Video Tutorials Website: CQPwebInABox Video Tutorials
Return to top.

Related posts on langui.ch:

AntConc on TurnKey Linux Server

If you try to launch AntConc on a Debian-based 64-bit system, you get the following error message (tested with versions 3.2.4u and 3.4.1u):

./antconc3.2.4u: No such file or directory
or 
./AntConc: No such file or directory

The following steps were necessary for me to be able to start AntConc on a TurnKey Linux Server (Debian7, 64-bit) using ssh with X11-forwarding enabled (e.g. PuTTY plus Xming on Windows 8.1).

Important note: Please respect Laurence Anthony’s licensing terms and ask for permission before using AntConc in a server/group environment (see README section ‘LEGAL MATTER’ (p. 11) for details).

1) Activate i386 architecture on 64-bit systems:

apt-get install libc6-i386
dpkg --add-architecture i386

2) Install missing 32-bit libraries:

apt-get install libx11-6:i386 libxss1:i386 libxft2:i386

For Ubuntu-based systems see this post, for other Linux distributions see ongoing discussion on: https://groups.google.com/forum/#!forum/antconc

AntWebCorpusFramework on Xubuntu 12.04 LTS

Steb-by-step guide to install Laurence Anthony’s AntWebCorpusFramework [1] on Xubuntu 12.04 LTS:

Requirements: webserver (e.g. apache2), php5, perl, parallel corpus (two sentence aligned parallel text files in strict utf-8 format)

[Click on 'Overview' and 'Select Category' ⇨ 'Corpus Development Tools' ⇨ 'Sentence Alignment' for a list of tools you could use to create your own parallel corpus.]

  1. Install apache2 web-server environment with php5 and enable sqlite support for perl and php5
  2. Extract files to a new AntWCF directory and create database(s)
  3. Adapt antpwc_concordance.php to new corpus
  4. Adapt index.php to new corpus
  5. Create apache2 configuration file
  6. Start webbrowser and go to http://localhost/AntWCF

Install apache2 web-server environment with php5 and enable sqlite support for perl and php5

Open a terminal window and type:

sudo apt-get install apache2 php5 php5-sqlite libdbd-sqlite3-perl

Troubleshooting:

If you would like to get rid of apache2‘s warning “Could not reliably determine the server’s fully qualified domain name”, click here for a quick fix (the server works just fine even if you don’t specify ServerName).

Open zip-file (available on request: [1]), extract all files into a new AntWCF directory and create database(s)

Change into the new AntWCF directory:

cd path/to/AntWCF

Troubleshooting:

If you get a permission error when trying to drag&drop files into the new AntWCF directory, you probably created the directory outside your home area, using the sudo command (e.g. in /opt). To be able to drag&drop files into the directory, you can take ownership by typing:

sudo chown -R yourusername:yourusername /path/to/AntWCF

After successfully extracting the files into your  AntWCF directory, copy your parallel corpus files to AntWCF/data/corpus.

To create a database for a parallel corpus type:

perl ./bin/db_creator_L8-CEN-R8da.pl data/corpus/L1_FILE.txt data/corpus/L2_FILE.txt data/AntWCF_db/OUTPUT_FILE.db

 Note: [1 corpus = 2 txt files (your files) + 1 db file (created by script)]

L1_FILE.txt is the source text file of your parallel corpus that you have just copied into the AntWCF/data/corpus directory, L2_FILE.txt is the target text file of your parallel corpus and OUTPUT_FILE.db is the name of the sqlite database to be created by the script (extension: .db).

Take a note of the name of the new database file and the number of tokens for L1/L2 displayed by the script (or make sure that you do not close the terminal window, as you will need these pieces of information later on).

Delete the temporary files created by the script:

rm temp_*

Repeat this procedure for other parallel corpora you might wish to include in AntWCF

Troubleshooting:

If the script throws an encoding error, make sure that your text files only contain legal utf8 characters and eliminate all non-utf8 characters before running the database script again. Important: If you choose the same output  file name, you have to delete the old file before re-running the script.

Adapt antpwc_concordance.php to new corpus

Open the file www/antpwc_concordance_20120925_2342.php in a text editor of your choice (e.g.):

leafpad www/antpwc_concordance_20120925_2342.php

[Select 'Options' ⇨  'Line Numbers' for easier navigation]

Adapt the following lines for your own parallel corpus:

Note: AntWCF is preconfigured to be able to switch between two different parallel corpara. If you created just one database file, comment out lines 32-34 and lines 40-42 (using ‘//‘ at the beginning of each line) or fill in the same data twice. If, however, you created more than two databases, you could include those corpora by copying lines 32-34, inserting them once before line 35 and a second time before line 43. Subsequently, you could adapt the database information for an additional corpus (and so on …). For number of tokens per language, refer back to the information generated by the db_creator.pl script above.

28-01-2014 10-42-53

Adapt index.php to new corpus

Open the file www/index.php in a text editor of your choice.

Adapt the following lines for your own parallel corpus:

Set default database:

29-01-2014 09-15-55

Adapt database information:

Note: AntWCF is preconfigured to be able to switch between two different parallel corpara. If you just created one database file, comment out line 110 or just fill in the same data twice. If you created more than two database files, copy line 110, insert it before line 111 and adapt the database file and corpus names:

28-01-2014 23-35-55

Adapt the language pair for your own parallel corpus (line numbers will have changed slightly if you inserted additional databases above).

28-01-2014 11-14-20

Create apache2 configuration file

Create an new file called AntWCF.conf in the directory /etc/apache2/conf.d/ and open it in a text editor of your choice (e.g.):

sudo leafpad /etc/apache2/conf.d/AntWCF.conf

Copy-paste the following lines into the new file, adapt the absolute path to your AntWCF/www directory (twice for Alias and for Directory) and save the file (you will need superuser privileges (sudo) to be able to save the file in this location):

# AntWCF default Apache configuration

Alias /AntWCF /path/to/AntWCF/www

<Directory /path/to/AntWCF/www>
     Options FollowSymLinks
     DirectoryIndex index.php
</Directory>

After saving the file successfully, restart apache2 by typing:

sudo service apache2 restart

Start webbrowser, go to http://localhost/AntWCF and enjoy!

29-01-2014 09-38-31

References:

  1. Anthony, L. (2014). AntWebCorpusFramework (Version 1.0) [Computer Software]. Tokyo, Japan: Waseda University. Available on request at http://www.laurenceanthony.net/

AntConc 3.4.1m (2014) on OSX Mavericks (OS X 10.9)

[Last update: 06/02/2014]

By default, if you try to open AntConc 3.4.1m (2014) on Mac OS X Mavericks, it is blocked by OSX Gatekeeper:

24-01-2014 00-32-47

  • To be able to start the app, you need to add it as an exception: Go to ‘Apple Icon’ ⇨ ‘System Preferences’ ⇨ ‘Security & Privacy’:

24-01-2014 00-37-03

  • Click on ‘Open Anyway’ (you need to have double-clicked AntConc before for this option to appear):

24-01-2014 00-38-22

  • and click on ‘open’ in the following dialogue:

24-01-2014 00-40-52

25-01-2014 12-53-06

 

For a discussion of known issues, visit: https://groups.google.com/forum/#!forum/antconc

Users reported:

  • screen refresh issues when changing parameters (workaround: change tab or move window to reflect changes)

AntConc 3.2.4m (2011) on OSX Mavericks (OS X 10.9)

There are two things that you have to bear in mind, when trying to start AntConc 3.2.4m on Mac OS X Mavericks:

(jump to Troubleshooting section if antconc pops up briefely and disappears again)

1) By default, if you try to open the app, it is blocked by OSX Gatekeeper:

23-01-2014-22-57-48.png

  • To be able to start the app, you need to add it as an exception: Go to ‘Apple Icon’ ⇨ ‘System Preferences’ ⇨ ‘Security & Privacy’:

23-01-2014 22-58-40

  • Click on ‘Open Anyway’ (you need to have double-clicked antconc3.2.4m before for this option to appear):

23-01-2014 22-59-02

  • and click on ‘open’ in the following dialogue:


23-01-2014 23-48-38

2) Download, install & run the latest X11:

  • The first time you try to open the app, you’ll see the following message:

23-01-2014 23-51-26

  • You can click on ‘Continue’ or go directly to the download page:
  • Download and open the latest .dmg file
  • Install XQuartz on your system
  • Start X11 (‘Finder’ ⇨ ‘Go’ ⇨ ‘Utilities’ ⇨ ‘XQuartz’)
  • Double-click antconc3.2.4m


24-01-2014 00-07-09

Troubleshooting

If the antconc3.2.4m icon only pops up briefly in the doc area and disappears again, you forgot to start X11 before starting AntConc:

Open ‘XQuartz’: ‘Finder’ ⇨ ‘Go’ ⇨ ‘Utilities’ ⇨ ‘XQuartz’

24-01-2014 00-09-06

and double-click antconc afterwards. (Go back to Section 2 if ‘XQuartz’ is not present in the ‘Utilities’ folder.)

 

Installing uplug on Linux

Tested on: Xubuntu 12.04 LTS (64-bit) for languages English, German (& French)

  • Install/check for uplug dependencies (libmodule-install-perl is compulsory, others could also be installed through cpan when running make):
  1. sudo apt-get install libmodule-install-perl libxml-parser-perl libxml-simple-perl libfile-sharedir-perl libipc-run-perl
  • Clone latest version from bitbucket repository:
  1. git clone https://bitbucket.org/tiedemann/uplug.git uplug
  • Third-party tools that are not automatically installed:
    • MElt (Tagger for French)
      • Dependencies: sudo apt-get install gawk python-numpy
  • Install individual modules:
cd uplug-XY
perl Makefile.PL
make
sudo make install
  • Install all modules:
cd uplug
make all
sudo make install
make test

Running InterText Editor on Linux

Tested on: Xubuntu 12.04 LTS (64-bit)

  1. sudo apt-get install libqt4-gui
  • Download latest release (InterText is a 64-bit linux binary without extension) and run InterText Editor:
  1. mkdir InterText
    cd InterText
    wget http://wanthalf.saga.cz/InterText
    chmod +x InterText
    ./InterText
  • Download icon and create desktop shortcut (copy-paste [Desktop Entry] into a new file called InterText.desktop on your Desktop, don’t forget to adapt the full path to InterText binary and InterText_logo.gif):
  1. wget http://langui.ch/wp/wp-content/uploads/2014/01/InterText_logo.png
  1. [Desktop Entry]
    Version=1.0
    Type=Application
    Name=InterText 1.1.1
    Comment=Parallel Alignment Editor 
    Exec=/path/to/InterText/InterText
    Icon=/path/to/InterText/InterText_logo.png