Countly Documentation

Countly Resources

Here you'll find comprehensive guides to help you start working with Countly as quickly as possible.

Troubleshooting

This document helps you troubleshoot a Countly instance, especially when you have a problem with Linux instance, MongoDB, processes or Countly application service itself.

Checking disk space

df -kh

Checking resource usage

#overall
vmstat 1

#memory
free -m

#by process
top

Checking if process was killed by system

Sometimes you can't find in the logs why process was killed, as it just exited. You can check here if process was killed because, for example, server run out of memory

dmesg | tail

#or

tail /var/log/kern.log -n 1000

#or 

tail /var/log/syslog -n 1000

Getting Countly version

countly version

Or if command line does not work

#run inside countly dir
cat package.json

Getting Countly installation directory

countly dir

or switching to Countly directory

cd `countly dir`

if Countly command line does not work

#if running on systemd
cat /etc/systemd/system/countly.service

#if running on upstart
cat /etc/init/countly-supervisor.conf

Enabling Countly command line if it's not available

#run inside countly dir
bash bin/scripts/detect.init.sh

Checking if Countly is running

countly status

Checking Countly logs

#run inside countly dir

#get latest 500 lines of api log
tail log/countly-api.log -n 500

#get latest 500 lines of dashboard log
tail log/countly-dashboard.log -n 500

Restarting Countly

#simply restart process
countly restart

#regenerate production files, install mising dependencies and restart process
countly upgrade

Checking and removing ghost processes

In rare cases, if a Countly process is stuck and you may want to kill and restart Countly.

# check current processes (should contain node or nodejs processes for countly)
top

#stop countly
countly stop

#recheck if still has node or nodejs processes
top

#if has them, kill them
pkill node

#start countly
countly start

Checking Countly http access without nginx

#checking if dashboard process runs (should return webpage)
wget -O- 127.0.0.1:6001

#checking if api process runs (should return 400 Bad request)
wget -O- 127.0.0.1:3001/i

Checking port connections

#checking amount of connections to mongodb (assuming default port 27017)
sudo netstat -nputw | grep 27017 | wc -l

Checking connection to database

#connects to db, gets document with configurations and outputs them
countly config list values

Increasing Countly limits

Limits for the process are managed by init system, so you need to change values in their config files which are

#if running on systemd
cat /etc/systemd/system/countly.service

#if running on upstart
cat /etc/init/countly-supervisor.conf

Adding a user to dashboard

#adds global admin from command line
countly add_user <username> <password>

#and then when not needed anymore to remove
countly remove_user <username> <password>

Clearing failed login time

#connect to mongo
mongo

#or use required parameters based on your mongodb setup

use countly
db.failed_logins.remove({_id:"<username>"})

Checking MongoDB version

mongod --version

Checking if mongod process is running

It should also reveal path to config file.

ps -ax | grep mongo

Starting mongod process

#starting MongoDB

#systemd
systemctl start mongod

#upstart
start mongod

#centos/rhel 6
service mongod start

Restarting MongoDB process

#restarting MongoDB on

#systemd
systemctl restart mongod

#upstart
restart mongod

#centos/rhel 6
service mongod restart

Checking MongoDB logs

#default log path is /var/log/mongodb/mongod.log or check in mongod.conf file for systemLog.path
#config file is usually at/etc/mongod.conf or check if process running to get path
tail /var/log/mongodb/mongod.log -n 500

Connecting to MongoDB

In simple case if MongoDB is on the same server its just

#In simple case if mongodb is on the same server its just
mongo

#if port is changed to for example 111222
mongo --port 111222

#if it is on another server, for example 192.168.3.100
mongo --host 192.168.3.100

#if it is behind authentication
mongo --username <username> --password <password> --authenticationDatabase <dbname>

#if it is through ssl
mongo --ssl --sslAllowInvalidCertificates

#if it is replica set
mongo --host <replSetName>/<hostname1><:port>,<hostname2><:port>,<...>

Checking how Countly connects to MongoDB

#run on countly server inside countly directory
cat api/config.js

Checking current/allowed connections

#connect to mongodb
mongo

#or use some correct approach depending on your mongodb configuration

#output connections info
db.serverStatus().connections

Checking current reads and writes

mongotop

#or use correct params (from connecting to mongodb) based on your mongodb configuration
#press Ctrl+C to exit

If reads and writes are quite long (longer than couple of seconds), your MongoDB instance might be overutilized and need to be scaled.

Checking currently used resources

mongostat

#or use correct params (from connecting to mongodb) based on your mongodb configuration
#press Ctrl+C to exit

Checking replica set status

#connect to mongodb
mongo

#or use some correct parameters depending on your mongodb configuration

#output replica set status
rs.status()

Checking process limits

First you need to create script that checks the limits

#!/bin/bash
if [ "$#" -ne "1" ]; then
   echo ""
   echo -e "\033[01;32mLimit checker\033[00m"
   echo -e "\033[01;37mUsage:\033[01;33m $0 process_name\033[00m"
   echo ""
   exit 0
fi

return-limits(){
   for process in $@; do
      process_pids=`ps -C $process -o pid --no-headers | cut -d " " -f 2`

      if [ -z $@ ]; then
         echo "[no $process running]"
      else
         for pid in $process_pids; do
            echo "[$process #$pid -- limits]"
            cat /proc/$pid/limits
      done
      fi
   done
}

return-limits $1

Then you can use it as

mongoserver041 ~ $ ./limits.sh mongod
[mongod #43447 -- limits]
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            10485760             unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             32000                32000                processes
Max open files            1000000              1000000              files
Max locked memory         65536                unlimited            bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       125029               125029               signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us

Increasing MongoDB limits

Limits for the process are managed by init system, so you need to change values in their config files which are

#if running on systemd
cat /etc/systemd/system/mongod.service

#if running on upstart
cat /etc/init/mongod.conf

Debugging and troubleshooting incoming SDK requests

If you would like to understand why data is not arriving to Countly server, or why you see partial data, check this FAQ item to troubleshoot further.

Checking if 3rd party email service functions correctly

  1. Got to Management > Email reports and create a new report.
  2. Press preview button, to make sure there is data to report and all is showing correctly.
  3. Optionally, clear the logs in Countly, specifically the countly/log/countly-api.log file.
    4) Press send now to send report to your provided email address.
    5) If no error in the log, it usually means e-mail has left the server. It is not possible for us to track it further.
    6) Check if the e-mail has arrived. There may be cues in your 3rd party service about this particular transaction.