ScopTEL Troubleshooting

Module 18 - ScopTEL IP PBX Software - Troubleshooting



General Troubleshooting


Most telephony errors are caused by one of the following causes:

Bad authentication credentials

DNS resolution

Class of Service misconfigurations

Firewall issues

Asterisk bugs

User error (DND, call forwarding etc.)


CLI tools are used to debug most of these issues WindowsTM Utilities:

Putty (WindowsTM based SSH tool)  http://the.earth.li/%CB%9Csgatham/putty/latest/

WinSCP (WindowsTM based, Linux file system editor and transfer client) http://winscp.net/download/


Log files

Server Logs: /var/log/messages

SMTP Logs: /var/log/maillog

Hardware Logs: dmesg

Asterisk Logs: /var/log/asterisk/messages

Asterisk Files: /etc/asterisk

ScopServ PHP logs: /tmp/php2.log

Asterisk Core Dumps (GDB files): dir /tmp/core*

SIP Response Codes 3XX

Redirection 3xx

300 Multiple Choices
301 Moved Permanently
302 Moved Temporarily
305 Use Proxy
380 Alternative Service

SIP Response Codes 4XX

Request Failure 4xx

400 Bad Request
401 Unauthorized
402 Payment Required
403 Forbidden
404 Not Found
405 Method Not Allowed
406 Not Acceptable
407 Proxy Authentication Required
408 Request Timeout
410 Gone
412 Conditional Request Failed [RFC3903]
413 Request Entity Too Large
414 Request-URI Too Long
415 Unsupported Media Type
416 Unsupported URI Scheme
417 Unknown Resource-Priority [RFC4412]
420 Bad Extension
421 Extension Required
422 Session Interval Too Small [RFC4028]
423 Interval Too Brief
424 Bad Location Information [RFC6442]
428 Use Identity Header [RFC4474]
429 Provide Referrer Identity [RFC3892]
430 Flow Failed [RFC5626]
433 Anonymity Disallowed [RFC5079]
436 Bad Identity-Info [RFC4474]
437 Unsupported Certificate [RFC4474]
438 Invalid Identity Header [RFC4474]
439 First Hop Lacks Outbound Support [RFC5626]
440 Max-Breadth Exceeded [RFC5393]
469 Bad Info Package [RFC6086]
470 Consent Needed [RFC5360]
480 Temporarily Unavailable
481 Call/Transaction Does Not Exist
482 Loop Detected
483 Too Many Hops
484 Address Incomplete
485 Ambiguous
486 Busy Here
487 Request Terminated
488 Not Acceptable Here
489 Bad Event [RFC6665]
491 Request Pending
493 Undecipherable
494 Security Agreement Required [RFC3329]

SIP Response Codes 5XX

Server Failure 5xx

500 Server Internal Error

501 Not Implemented

502 Bad Gateway

503 Service Unavailable

504 Server Time-out

505 Version Not Supported

513 Message Too Large

580 Precondition Failure                  [RFC3312]

SIP Response Codes 6XX

Global Failures 6xx

600 Busy Everywhere

603 Decline

604 Does Not Exist Anywhere

606 Not Acceptable

OS CLI Commands

scopserv_yum update
executes a full software update
scopserv_yum install <package name>
installs a package from repositories
scopserv_dumpkey
displays license information
dkms status
shows install status of dahdi wanpipe mISDN drivers
asterisk –r
opens Asterisk CLI for Asterisk command line debugging
asterisk –vr 
opens Asterisk CLI for Asterisk command line debugging with increased verbosity
lsof -i :5555
shows which service is using port 5555
rpm –qa <package name>
shows package version
rpm –e <package name> --nodeps
removes package without uninstalling dependencies
rpm –qa |grep <package name>
shows installed packages version
chown –R scopserv:scopserv /<path>
changes ownership recursively on a path
kill –KILL <pid>
kills a process ID
htop
shows all processes and consumption
top
shows all processes and consumption
ngrep -d any port 25 -W BYLINE
network trace on port 25 to debug SMTP issues
network trace on port 25 to debug SMTP issues
Linux DNS lookup for MX record on specified domain
rm -rf /root/.ssh/known_hosts 
rm -rf /root/.ssh/known_hosts 
wget <package path>
CLI command to download a file from path
ngrep –d port any 69 
network trace on port 69 to display TFTP traffic
cat /<path>
display contents of file to screen
nano /<path> 
Linux text editor to edit path
service <service name> restart
restart network without rebooting server
service <service name> stop
stop service name without rebooting server
service <service name> status
display service name status
reboot
reboot server
shutdown -h now
shutdown server
lynx <website address> 
Linux command line web browser
lynx <website address> 
Ping an IP address
lynx <website address> 
trace an IP address through multiple hops
tcpdump -nq -s 0 -i eth0 -w /tmp/sip.pcap port not ssh
saves a SIP pcap trace to /tmp using eth0
ps auwwwwx |grep crond 
prints all crond PID’s
rpm -e --allmatches <package name> –nodeps 
`removes the package name without dependencies
rpm -e <package name> --nodeps –noscripts
removes a broken package forcibly
route
prints the routing table
Ifconfig 
The "ifconfig" command allows the operating system to setup network interfaces and allow the user to view information about the configured network interfaces. 
nmtui
Command line utility to configure network interfaces

Asterisk CLI Commands

To activate enter asterisk –r from Linux CLI
core show channels
display active channels
sip show peers 
show all SIP peers status
sip show peer <peer>
show details of SIP peer
database show 
display  astDB info (useful to see if a phone is in DND status
queue show 
displays all ACD queues info
pri show spans
displays status of all PRI spans
pri intense debug span <span number>
shows Q.931 and SABME messages
pri set debug off span <span number 
stop intense PRI debugging on span number
hangup request <channel>|<all>
Request that a channel be hung up. The hangup takes effect the next time the driver reads or writes from the channel. If 'all' is specified instead of a channel name, all channels will see the hangup request.
core show hints
This command lists registered hints
agi set debug on
prints agi debug messages (always useful when creating a support ticket)
sip set debug <on/off>
prints SIP messages to the CLI


Database Repair Commands

mysqlcheck –A –r
Automatically optimizes and repairs mysql tables. If crashed on Slave (HA) then this need to be done on slave and not master 
service scopserv stop
Stops the GUI service and realtime
lsof -i :5555
Displays the PID that is bound to the scopserv service port
kill –KILL <PID output of lsof –I 5555>
Kills the PID bound to the scopserv port
service scopserv start
Starts the GUI once the binding port is freed

MySQL Repair is Taking Too Long

If the Server is not configured in High Availability Mode (standard installation):
mysql scopserv (enter)
truncate cel; truncate phones_history; truncate horde_histories;
truncate cel; truncate phones_history;
truncate horde_histories;

If the Server is configured in High Availability Mode:
mysql scopserv_repl (enter)

ScopTEL GUI is Slow

mysqlcheck –A –r : Automatically optimizes and repairs mysql tables. If crashed on Slave (HA) then this need to be done on slave and not master 
scopserv_yum update: Older packages, especially prior to scopserv_realtime nodejs changes are not fully optimized
Telephony>Configuration>Reports (CDR/ACD)>Archiving>Enable Archiving :[x]


ScopTEL GUI Will Not Restart

mysqlcheck –A –r
 Automatically optimizes and repairs mysql tables. If crashed on Slave (HA) then this need to be done on slave and not master 
service scopserv stop
 Stops the GUI service and realtime
lsof -i :5555
 Displays the PID that is bound to the scopserv service port
kill –KILL <PID output of lsof –I 5555>
 Kills the PID bound to the scopserv port
service scopserv start
 Starts the GUI once the binding port is freed

ScopTEL GUI Fatal Error and Dead Call Processing
You must delete files on the hard drive to restore call processing
Usually this is caused by:
Too many local backups
Too many recording files in /var/spool/asterisk/monitor/
Too many files in /var/log/asterisk/
Too many files in /tmp/
To locate the largest folders do:
cd / Change to root directory
du * -s Print Disk Usage Summary  and locate the largest folder
Use the ‘cd’ command to change into the largest directory
Use the Linux ‘rm’ command to remove files
Keep drilling down from / and use the ‘du * -s’ command to locate the largest folders and keep deleting files with the ‘rm’ command until you have freed enough space in the file system to restore normal service.

Debugging Authentications Failures

Here is a typical authentication failure in Asterisk. The reason for this failure is that peer 228 does not exist!

[root@demo ~]# asterisk -vr

=========================================================================

[Aug 1 14:41:46] Running as group 'scopserv'

[Aug 1 14:41:46] Connected to Asterisk 1.8.12.0 currently running on demo (pid = 6006)

Verbosity is at least 3

Core debug is at least 3

[2012-08-01 14:50:06] NOTICE[6260]: chan_sip.c:24974 handle_request_register: Registration from '"228"<sip:228@scopserv.local>' failed for '192.168.192.55:57744' - No matching peer found



Here is a typical authentication failure in Asterisk. The reason for this failure is that the password does notmatch!


[root@demo ~]# asterisk -vr

=====================================================================================

[Aug 1 14:41:46] Running as group 'scopserv'

[Aug 1 14:41:46] Connected to Asterisk 1.8.12.0 currently running on demo (pid = 6006)

Verbosity is at least 3

Core debug is at least 3

[2012-08-01 14:53:49] NOTICE[6260]: chan_sip.c:24974 handle_request_register: Registration from '"227"<sip:227@scopserv.local>' failed for '192.168.192.55:63316' - Wrong password



Here is a typical authentication success in Asterisk


[root@demo ~]# asterisk -vr

==========================================================

[Aug 1 14:41:46] Running as group 'scopserv'

[Aug 1 14:41:46] Connected to Asterisk 1.8.12.0 currently running on demo (pid = 6006)

Verbosity is at least 3

Core debug is at least 3

[2012-08-01 14:56:00] NOTICE[6260]: chan_sip.c:20812 handle_response_peerpoke: Peer '227' is now Reachable. (7ms / 2000ms)


Debugging Call Failures With ‘database show’

In this first example a call from 227 to 221 fails to ring 221

[2012-08-01 15:02:05] == Extension Changed 227[default-local] new state InUse for Notify User 221

[2012-08-01 15:02:05] -- Executing [221@default-default:9] Macro("SIP/227-00000005", "defaultdial,SIP/221,221,default,,en,u221@default,twWxXkKg,,default_default,,Local/0@default-local/n,vm") in new stack

[2012-08-01 15:02:05] -- Executing [s@macro-default-dial:1] NoOp("SIP/227-00000005", ""CALL TO LOCAL EXTENSION FROM 227(227)"") in new stack

[2012-08-01 15:02:05] -- <SIP/227-00000005>AGI Script agi://127.0.0.1:4573/dial completed, returning 0

[2012-08-01 15:02:05] -- Executing [s@macro-default-dial:4] ExecIf("SIP/227-00000005", "1?Macro(all-vm,default,b,221@default,Local/0@defaultlocal/n,vm)") in new stack

[2012-08-01 15:02:05] -- Executing [s@macro-all-vm:1] NoOp("SIP/227-00000005", "CALL TO VOICEMAIL") in new stack • [2012-08-01 15:02:05] -- Executing [s@macro-all-vm:2] VoiceMail("SIP/227-00000005", "221@default,b") in new stack

database show the extension has enabled the DND feature code!

demo*CLI> database show

/CustomDevstate/dnd-default-221 : BUSY

/DND/Local/*78@default                 : On

/DND/SIP/221                                      : On


Debugging Call Failures with SIP Cause Codes

In this second example a call from 227 to 221 fails to ring 221

[2012-08-01 15:24:27] -- Executing [221@default-default:1] Set("SIP/227-00000009", "CDR(tenant)=default") in new stack

[2012-08-01 15:24:27] -- Executing [221@default-default:9] Macro("SIP/227-00000009", "defaultdial,SIP/221,221,default,,en,u221@default,twWxXkKg,,default_default,,Local/0@default-local/n,vm") in new stack

[2012-08-01 15:24:27] -- Executing [s@macro-default-dial:1] NoOp("SIP/227-00000009", ""CALL TO LOCAL EXTENSION FROM 227(227)"") in new stack

[2012-08-01 15:24:27] -- Called SIP/221

[2012-08-01 15:24:27] -- Got SIP response 302 "Moved Temporarily" back from 192.168.100.100:5060

[2012-08-01 15:24:27] -- Now forwarding SIP/227-00000009 to 'Local/555@default-default' (thanks to SIP/221-0000000a)

[2012-08-01 15:24:27] NOTICE[30807]: app_dial.c:892 do_forward: Not accepting call completion offers from call-forward recipient Local/555@default-default-59dd;1

[2012-08-01 15:24:27] NOTICE[30807]: chan_local.c:899 local_call: No such extension/context 555@default-default while calling Local channel

[2012-08-01 15:24:27] NOTICE[30807]: app_dial.c:980 do_forward: Forwarding failed to dial 'Local/555@default-default'

[2012-08-01 15:24:27]  == Everyone is busy/congested at this time (1:0/0/1)

The SIP response code shows the extension has enabled a native SIP forward to 555 from the SIP phone!  There is no 555 in the dial plan and the forward fails.  The user must remove the bad forwarding configuration from their phone.


ScopSTATS System Monitoring Logs

​The ScopSTATS reporting engine can also display System Logs instead of accessing the Linux file system. Use the various drop list selectors to generate reports.







    • Related Articles

    • ScopTEL IP PBX Software - Basic Installation Hierarchy for Telephony Server

      Basic Installation Hierarchy for Telephony Server Therefore the purpose of this document is to provide a visual walkthrough of a very basic but functional installation for one tenant. This tutorial does not include an overview of the overall network ...

    • Module 14 - ScopTEL IP PBX Software - Managing ACD

      Automatic Call Distribution (ACD)  | Background Automatic Call Distribution queues put callers into a queue and typically play Music on Hold or custom announcement messages to keep them informed or relaxed while they wait for an agent to become ...

    • Module 8 - ScopTEL IP PBX Software - Extensions Management

      Security | Background SIP Phones are SIP User Agents. For security, SIP User Agents must register to the SIP Registrar via username and password authentication. It is typical for the SIP protocol ports to be open or forwarded to the ScopTEL server if ...

    • Module 19 - ScopTEL IP PBX Software - Quality-of-Service-Training

      The use of IP as a foundation for converged networks has raised several issues for both enterprise IT departments and ISPs.  IP and Ethernet are connectionless technologies and do not guarantee bandwidth. The protocol will not differentiate network ...

    • Module 4 - ScopTEL IP PBX Software - PSTN Interfaces and Gateways

      Gateways A VoIP gateway is as a bridge between: Interfaces:  (T1/E1, FXO, FXS) Protocols:  SIP, Cisco SCCP/Skinny, MGCP (legacy protocol), and H.323 (legacy protocol) CODECS (media):  GSM (13 Kbps), iLBC (15 Kbps), G.711 (64 Kbps), G.722 (48/56/64 ...