Erlang/OTP Forums

Author Message

<  Ejabberd mailing list  ~  ejabberd stability problem

Guest
Posted: Mon Dec 04, 2006 5:23 am Reply with quote
Guest
Dear friends,

have problem with ejabberd server stability. Ejabberd is crashing every
2-6 hours.
Max 200 users are online.
PyICQt and PyMSNt are running.

I have ejabberd 1.1.2 installed from source installed on Fedora FC4
erlang-R10B-10.1.fc4

Starting command:
exec erl -shared -env ERL_MAX_PORTS 32000 -pa /var/lib/ejabberd/ebin
-sname ejabberd -s ejabberd -ejabberd config
\"/etc/ejabberd/ejabberd.cfg\" log_path \"/
var/log/ejabberd/ejabberd.log\" -sasl sasl_error_logger
\{file,\"/var/log/ejabberd/sasl.log\"\} -mnesia dir
\"/var/lib/ejabberd/spool\" -detached
or
exec erl -env ERL_MAX_PORTS 32000 -pa /var/lib/ejabberd/ebin -sname
ejabberd -s ejabberd -ejabberd config \"/etc/ejabberd/ejabberd.cfg\"
log_path \"/
var/log/ejabberd/ejabberd.log\" -sasl sasl_error_logger
\{file,\"/var/log/ejabberd/sasl.log\"\} -mnesia dir
\"/var/lib/ejabberd/spool\" -detached


This is typical crash dump report:
=erl_crash_dump:0.1
Sun Dec 3 13:45:54 2006
Slogan: no next heap size found: 145975097, offset 0
System version: Erlang (BEAM) emulator version 5.3.6.3 [source] [hipe]
[shared heap]
Compiled: Wed Apr 12 20:59:59 2006
Atoms: 9115
=memory
total: 1681327485
processes: 1169104474
processes_used: 1169092858
system: 512223011
atom: 414389
atom_used: 397731
binary: 640125
code: 3666715
ets: 499325592
=hash_table:atom_tab
size: 6421
used: 4825
objs: 9115
depth: 7


Typical memory usage:
top - 16:51:57 up 2:13, 1 user, load average: 0.06, 0.13, 0.17
Tasks: 129 total, 1 running, 126 sleeping, 0 stopped, 2 zombie
Cpu(s): 9.6% us, 2.7% sy, 0.0% ni, 82.1% id, 5.6% wa, 0.0% hi, 0.0% si
Mem: 905420k total, 861348k used, 44072k free, 8564k buffers
Swap: 3349124k total, 824k used, 3348300k free, 161716k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2374 root 15 0 77268 11m 2684 S 5.6 1.3 0:19.90 python
2385 root 15 0 23916 10m 2988 S 2.0 1.2 0:06.36 python
2523 root 15 0 602m 567m 2684 S 1.3 64.2 3:47.30 beam.shared
2896 root 15 0 5696 2496 1380 S 0.3 0.3 0:20.49 ssl_esock
1 root 16 0 1748 572 492 S 0.0 0.1 0:01.37 init
2 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0
3 root RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
4 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 events/0
5 root 16 -5 0 0 0 S 0.0 0.0 0:00.01 khelper
6 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kthread

ejabber.cfg (part)

% Listened ports:
{listen,
[
{5222, ejabberd_c2s, [{access, c2s}, {shaper, c2s_shaper}]},
{5223, ejabberd_c2s, [{access, c2s}, ssl, {certfile, "./ssl.pem"}]},
{5269, ejabberd_s2s_in, [{shaper, s2s_shaper}]},
{5280, ejabberd_http, [http_poll, web_admin]},
{5304, ejabberd_service, [{access, all},
{host, "icq.jabber.sk",
[{password, "XXXXXX"}]}]},
{5305, ejabberd_service, [{host, "msn.jabber.sk",
[{password, "XXXXXXX"}]}]},
{5301, ejabberd_service, [{host, "aim.jabber.sk",
[{password, "XXXXXXXX"}]}]}
]}.


% Use STARTTLS+Dialback for S2S connections
{s2s_use_starttls, true}.
{s2s_certfile, "./ssl.pem"}.

{outgoing_s2s_port, 5269}.


% Used modules:
{modules,
[
{mod_register, [{access, register}]},
{mod_roster, []},
{mod_privacy, []},
{mod_configure, []},
{mod_configure2, []},
{mod_disco, []},
{mod_stats, []},
{mod_vcard, []},
{mod_offline, []},
{mod_announce, [{access, announce}]},
{mod_echo, [{host, "echo.localhost"}]},
{mod_private, []},
{mod_irc, []},
{mod_muc, [{access, muc},
{access_create, muc},
{access_admin, muc_admin}]},
{mod_pubsub, []},
{mod_time, []},
{mod_last, []},
{mod_version, []}
]}.



I have no idea what could be the problem.
Any ideas ?

Thank you

Michael Hora




_______________________________________________
ejabberd mailing list
ejabberd@jabber.ru
http://lists.jabber.ru/mailman/listinfo/ejabberd
Post recived from mailinglist
Guest
Posted: Mon Dec 04, 2006 8:37 am Reply with quote
Guest
Hello,
I would first avoid using shared heap.

I hope this helps,

--
Guest
Posted: Mon Dec 04, 2006 8:54 am Reply with quote
Guest
Hello,

see my original message. I am running ejabberd with or without shared
option.
Server is crashing with and without shared heap.

Michael

Starting command:
exec erl -shared -env ERL_MAX_PORTS 32000 -pa /var/lib/ejabberd/ebin
-sname ejabberd -s ejabberd -ejabberd config
\"/etc/ejabberd/ejabberd.cfg\" log_path \"/
var/log/ejabberd/ejabberd.log\" -sasl sasl_error_logger
\{file,\"/var/log/ejabberd/sasl.log\"\} -mnesia dir
\"/var/lib/ejabberd/spool\" -detached
or
exec erl -env ERL_MAX_PORTS 32000 -pa /var/lib/ejabberd/ebin -sname
ejabberd -s ejabberd -ejabberd config \"/etc/ejabberd/ejabberd.cfg\"
log_path \"/
var/log/ejabberd/ejabberd.log\" -sasl sasl_error_logger
\{file,\"/var/log/ejabberd/sasl.log\"\} -mnesia dir
\"/var/lib/ejabberd/spool\" -detached



Micka
Guest
Posted: Mon Dec 04, 2006 9:59 am Reply with quote
Guest
Hello,
Le 4 d
Guest
Posted: Mon Dec 04, 2006 11:49 am Reply with quote
Guest
Hello.

> could you check if the crash still happen without the gateways
> (turned of from config files) ?
all services I have turned off in config file

> By the way, could you also replace ssl keyword in your config file by
> tls ?

ssl -> tls changed
{5222, ejabberd_c2s, [{access, c2s}, {shaper, c2s_shaper}]},
{5223, ejabberd_c2s, [{access, c2s}, tls, {certfile, "./ssl.pem"}]},

> ssl implementation used to produced this type of problems.
>
This is situation 1 minute before crash:

top - 12:26:09 up 1:12, 1 user, load average: 1.31, 1.12, 0.66
Mem: 905420k total, 737188k used, 168232k free, 1944k buffers
Swap: 3349124k total, 639896k used, 2709228k free, 22068k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2550 root 20 0 1216m 620m 2020 R 94.7 70.2 6:33.43 beam
20275 root 16 0 2016 1016 784 R 0.3 0.1 0:00.33 top
1 root 16 0 1748 492 468 S 0.0 0.1 0:01.38 init
2 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0
3 root RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
4 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 events/0
5 root 15 -5 0 0 0 S 0.0 0.0 0:00.01 khelper
6 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kthread
8 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 kacpid
64 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kblockd/0
67 root 15 0 0 0 0 S 0.0 0.0 0:00.00 khubd
113 root 15 0 0 0 0 S 0.0 0.0 0:00.13 pdflush
114 root 15 0 0 0 0 S 0.0 0.0 0:00.21 pdflush
116 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 aio/0

ejabberd uptime less than 2 hours Sad

=erl_crash_dump:0.1
Mon Dec 4 12:26:16 2006
Slogan: no next heap size found: 154994461, offset 0
System version: Erlang (BEAM) emulator version 5.3.6.3 [source] [hipe]
Compiled: Wed Apr 12 20:59:59 2006
Atoms: 9621
=memory
total: 1186573096
processes: 671210650
processes_used: 671171002
system: 515362446
atom: 428973
atom_used: 420693
binary: 715966
code: 4014736
ets: 502379484

Any ideas ?

Michael



_______________________________________________
ejabberd mailing list
ejabberd@jabber.ru
http://lists.jabber.ru/mailman/listinfo/ejabberd
Post recived from mailinglist
Guest
Posted: Mon Dec 04, 2006 12:56 pm Reply with quote
Guest
Hello.

7 minutes before crash I have last record in sasl.log
I don't know what it is.

Michael

---------------------------

=SUPERVISOR REPORT==== 4-Dec-2006::12:19:33 ===
Supervisor: {local,ejabberd_http_sup}
Context: child_terminated
Reason: {{badmatch,{badrpc,{'EXIT',
{{badmatch,
{error,
systools_rc,
{function_clause,
[{systools_rc,
split_instrs,
[{application,
ejabberd,
permanent,
[],
[],
[],
[],
[],
[],
[],
[],
infinity,
infinity,
[],
[],
[]},
[]]},
{systools_rc,
split_script,
1},
{systools_rc,

'-merge_scripts/1-fun-0-',
2},
{lists,foldl,3},
{systools_rc,
merge_scripts,
1},
{systools_rc,

do_translate_scripts,
3},
{systools_rc,
translate_scripts,
3},
{ejabberd_update,

make_low_level_script,
2}|
more]}}},
[{ejabberd_update,
make_low_level_script,
2},
{ejabberd_update,update_info,0},
{rpc,call,4},
{ejabberd_web_admin,get_node,5},

{ejabberd_web_admin,process_admin,2},
{ejabberd_http,process_request,1},
{ejabberd_http,process_header,2},
{ejabberd_http,receive_headers,1}|
more]}}}},
[{ejabberd_web_admin,get_node,5},
{ejabberd_web_admin,process_admin,2},
{ejabberd_http,process_request,1},
{ejabberd_http,process_header,2},
{ejabberd_http,receive_headers,1},
{proc_lib,init_p,5}]}
Offender: [{pid,<0.3012.0>},
{name,undefined},
{mfa,{ejabberd_http,start_link,
[{gen_tcp,#Port<0.8284>},
[http_poll,web_admin]]}},
{restart_type,temporary},
{shutdown,brutal_kill},
{child_type,worker}]

_______________________________________________
ejabberd mailing list
ejabberd@jabber.ru
http://lists.jabber.ru/mailman/listinfo/ejabberd
Post recived from mailinglist

Display posts from previous:  

All times are GMT
Page 1 of 1
This forum is locked: you cannot post, reply to, or edit topics.

Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You can attach files in this forum
You can download files in this forum