Intermittent Phone Issues (Resolved)
  • Priority - Critical
  • Affecting System - voip
  • We have been having issues with out telephone system. You can always open a support ticket in your customer portal.

    www.zenithmedia.ca/portal/

    We are investigating and hope to have it resolved soon.

    Outgoing call are working; it seems to be an AUDIO issue. You can still call directly to your account managers extention.
    Or Press 1 for sales and 2 for support.

    You will not receive any indication untill someone picks up the line.

    Sorry for the inconveience.

    - Jon Wong

    --- Updated 02/06/2017

    The audio issue has been resolved and the full IVR is online. Sorry for the problem.

    The issue was with our upstream provider and it was related to a codex upgrade that scambled our IVR audio voiced by our lovely Kimmy Pops.

    If you have issues, please open a ticket.

  • Date - 24/01/2017 17:13 - 06/02/2017 05:26
  • Last Updated - 09/02/2017 17:53
Billing System Issues (Resolved)
  • Priority - Critical
  • Affecting System - portal
  • We are experiencing issues with out biling system; this issue is only effecting the creation of e-Mails.

    Some clients might not receive their invoice/quotes or account creation emails. We will do our best to manually send out the required information. We hope to resolve this issue as soon as possible.


    Thank you.

    - Jon Wong
    Founder of Zenith Media


    -- updated November 8th 2016

    We have resolved the issue, it was due to the code on our automatic login forms. We have adjusted our system and the change in code is confirmed working.

    As part of our effort to give back to the community; this cause can be viewed on our github page. www.zenithmedia.ca/github

  • Date - 06/11/2016 14:29 - 08/11/2016 15:13
  • Last Updated - 01/12/2016 10:02
anycast-dzone2 (Resolved)
  • Priority - Low
  • Affecting System - anycast02.zenithmedia.net
  • We have received reports that our status page located at https://www.zenithmedia.ca/uptime was showing anycast-dzone2 as down. We can confirm that this status is a false positive. We did not update the status page with the updated IP address.

    anycast02.zenithmedia.net is the offical hostname for this DNS cluster. This particular cluster is part of our partnership with Cira the registar responsible for the .CA domain names.

    No Domains / Hosting / eMail or DNS services were effected by this error.

    If for some reason you think this effected you, please open a ticket and we will investigate further.



  • Date - 05/11/2016 10:17 - 05/11/2016 00:00
  • Last Updated - 05/11/2016 10:22
Downtime BHS (Resolved)
  • Priority - Critical
  • Affecting Server - Caspar - WEBCA
  • A power outage was experienced at the BHS Data center taking down over 7,000 instances of the Public Cloud infrastructure.

    Power bay had a short circuit and fried the whole cuircut. Electrial teams were dispatched and ~2,500 servers are back online.

    All our client servers are back online as of 9:30 am.

    Affected services where websites only. Mail was received and queued on our Europe servers and are delivering to our clients as we write this.

    We are in the process of investigating load balancing and replication of our website infrasructure in hopes of providing realiable uptime to our clients.
    No data loss is expected and all system are up.

    Please report any problems, due to the nature of the problem and active electrical work being done. It is possible that the servers flap up and down a little.

    Here is the offical tweet from the Datacenter CEO - https://twitter.com/olesovhcom/status/706050081997332480

  • Date - 04/03/2016 21:51 - 05/03/2016 09:28
  • Last Updated - 30/03/2016 10:13
CVE-2015-7547 (Resolved)
  • Priority - Critical
  • Affecting Server - Caspar - WEBCA
  • CVE-2015-7547 is a critical vulnerability in glibc affecting any versions greater than 2.9. The DNS client side resolver function getaddrinfo() used in the glibc library is vulnerable to a stack-based buffer overflow attack. This can be exploited in a variety of scenarios, including man-in-the-middle attacks, maliciously crafted domain names, and malicious DNS servers.

    All servers have been updated to resolve this issue.

  • Date - 23/02/2016 04:52 - 23/02/2016 04:53
  • Last Updated - 23/02/2016 04:53
Mail server migration issue (Resolved)
  • Priority - Critical
  • Affecting Server - Caspar - WEBCA
  • We have noticed an issue during our last client migration; We found a bug in the migration script not setting email domains locally.
    We have resolved the issue and all emails are flowing normally. If you have any email issue related to this bug please open up a ticket (replies via facebook comments are not always monitored)

    If you reveiced the following errors;

    Your message cannot be delivered to the following recipients:

     Recipient address: you@yourdomain.com
     Reason: Remote SMTP server has rejected address
     Diagnostic code: smtp;550-Please turn on SMTP Authentication in your mail client. remote.email.server.tld [XXX.XXX.XXX.XXX]:43941 is not permitted to relay through this server without authentication.

     Remote system: dns;caspar.zenithmedia.net (TCP|XXXXXXXX|43941|158.69.73.233|25) (caspar.zenithmedia.net ESMTP Exim 4.86 #2 Sun, 10 Jan 2016 11:38:09 -0500 )


    The cause was resolved, please re-send any emails that could have failed due to this bug.

    Regards

  • Date - 10/01/2016 11:00 - 10/01/2016 12:20
  • Last Updated - 10/01/2016 12:24
Fiber cut at main Data Center (Resolved)
  • Priority - Critical
  • Affecting Server - MAGI - WEBCA
  • Our primary data center in Montreal, Quebec had a fiber optic cut in a tunnel 108km away and experienced 5 hours and 30 minutes of intteruption.
    All services are currently restored and everything is back online.

    Unfortunetlly this interruption was out of our control and due to the nature of it, our infrastructure was affected, we noticed issued with our backup mail servers and resolved them.

    We are currently investigating other backup options to mitigate these kinds of issues in the future.

    If you have any questions regarding this issue, please contact us using our Portal.


    -Jon Wong
    Founder of Zenith Media Canada

  • Date - 02/11/2015 11:30 - 02/11/2015 16:30
  • Last Updated - 05/12/2015 09:00
Change of HOSTNAME for magi.zenithmedia.me (Resolved)
  • Priority - Critical
  • Affecting Server - MAGI - WEBCA
  • We are in the process of changing the hostname of one of our web clusters, this change will have backward compatibility but we request all clients to update their settings.

    The hostname is in question is magi.zenithmedia.me its being updated to magi.zenithmedia.net to reflect our network domain. All other servers have been from the start on the .net domain. So no major changes should be expected.

    We are in the process of updating MX/TXT/SRV records for ALL client domains and all parent domain records that we manage, unfortunetlly we do not manage all domains for our clients and cannot force change to domains we don't manage. Hense why we would like to inform everyone of the changes.

    We will do our best to update all our clients who are effected by this change.

    Regards,

  • Date - 15/09/2015 02:56 - 15/09/2015 04:00
  • Last Updated - 18/10/2015 06:38
Maintenance of Failover (Resolved)
  • Priority - Critical
  • Affecting Server - MAGI - WEBCA
  • We performed failover of our web network this morning and encountered failures from Pingdom, the issue we discovered is that even if we failover correctly pingdom is testing via IP and not hostname.

    www.zenithmedia.ca/uptime will show a downtime of 3:45 for September 6th 2015.


    We will re-configure pingdom with the changes and our uptime stats should be cleared up to reflect our commitment to 100% uptime.

  • Date - 06/09/2015 05:15 - 06/09/2015 07:00
  • Last Updated - 18/10/2015 06:37
PHP 5.4.43 Patch (Resolved)
  • Priority - Medium
  • Affecting Server - MAGI - WEBCA
  • We are in the processes of updating all web servers to PHP 5.4.43 to address security issues reported by NIST.

    The National Vulnerability Database (NIST) has given the following severity ratings to these CVEs:

    CVE-2015-3152 – MEDIUM

    PHP 5.4.43
    Fixed bug in mysqlnd library related to CVE-2015-3152

    https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2015-3152
    http://php.net/ChangeLog-5.php

    --completed

  • Date - 15/07/2015 00:22 - 15/07/2015 01:14
  • Last Updated - 15/07/2015 02:23
Primary site & portal ip routing issue (Resolved)
  • Priority - Critical
  • Affecting Other - Primary Site www.zenithmedia.ca
  • We started investigating reports our primary website and portal system where offline, We contacted our upstream provider and noticed a routing issue after an upgrade to our cloud infrastructure.
    This issue did not effect our clients and secondary services (anycast,erp,mail) but this take our primary website and client portal system offline. We reported the issue and it was resolved. If you are reading this, then everything is working!

    All orders where suspended during this time. If you tried to place and order, please try again.


    If you experience any issues since this has been resolved, please open a support ticket.

    As of 15/07/2015 the issue has been resolved and confirmed stable.

  • Date - 09/07/2015 03:16 - 15/07/2015 00:00
  • Last Updated - 15/07/2015 00:50
Issue with GEO IP (Resolved)
  • Priority - High
  • Affecting Other - Primary Site www.zenithmedia.ca
  • We have had issues with our primary site loading extreamly slow.
    It seems to be an issue with our GEO IP provider Telize, we have switched to FreeGEOIP and the load has been removed.
    The issue seems to be with IPv6 resolution, we will investigate the issue and see what can be resolved.

    The GEOIP DB is used to generate our company phone numbers you see at the top/bottom of every page. We use this technique in hopes to provide the best contact number based on the location of our clients. Montreal, Toronto and Toll free are displayed depending on location.

    We will possibly remove this feature due to the loading issues we have had in the past.

    This feature was removed from www.zenithmedia.ca/portal/ due to above stability issues.

  • Date - 03/06/2015 08:37 - 05/12/2015 09:00
  • Last Updated - 10/07/2015 01:34
HeartBleed Bug (Resolved)
  • Priority - Critical
  • Affecting Server - MAGI - WEBCA
  • We have finalized our investigation on the "Heat Bleed" SSL bug and our systems are no longer effected.

    Based on our investigation no system was compromised.

    Here are the results from our tests.

    https://www.ssllabs.com/ssltest/analyze.html?d=www.zenithmedia.ca

    For more information on the SSL bug you can visit http://heartbleed.com/

  • Date - 11/04/2014 13:00 - 11/04/2014 15:00
  • Last Updated - 30/04/2015 18:01
DNS Change & Account Move. (Resolved)
  • Priority - Low
  • Affecting Server - MAGI - WEBCA
  • Changed the DNS settings on 10 legacy accounts and moved them to our new servers.

    No down time.
     

  • Date - 02/04/2013 08:58 - 02/04/2013 08:58
  • Last Updated - 12/02/2015 13:10
Apache 2.4.10 Update (Resolved)
  • Priority - High
  • Affecting Server - MAGI - WEBCA
  • We have updated Apache to version 2.4.10 as a system performance test. We have removed mod_pagespeed during the tests.
    If you rely on mod_pagespeed for advanced filtering, please contat us for alternate solutions.

    Apache performance improvement is geared towards HTTPS security.

    -- Update Jan 16th 2015

    We have re-enabled mod_pagespeed after initial testing. It will be activated by default if you had it previously enabled.

    You can enable advanced catching by adding this to your .htaccess

    <IfModule pagespeed_module>
    ModPagespeed on
    </IfModule>

    For advanced configuration and reference please contact your support representative.

  • Date - 10/01/2015 00:00 - 10/01/2015 01:00
  • Last Updated - 16/01/2015 18:21
Issue with .htaccess and Apache 2.4 (Resolved)
  • Priority - Critical
  • Affecting Server - MAGI - WEBCA
  • We have received reports of people seeing 500 error code when accessing their sites.
    The issue is with an option found in .htaccess SetEnv TZ Etc/GMT-5
    It is not compatible with Apache 2.4 and should be commented out. This will resolve any issue with the 500 error code you would see displayed.

    If you have any issues or are registered as a managed account please contact support. We have taken the initiative and resolved the issue with all our clients.

  • Date - 15/01/2015 18:08 - 16/01/2015 00:00
  • Last Updated - 16/01/2015 18:18
DKIM Issue (Resolved)
  • Priority - Critical
  • Affecting Server - MAGI - WEBCA
  • We received reports of an error with our DKIM validation. Our primary mail server rejected valid DKIM signatures.
    We found the bug in an updated ACL rule, the issue has been resolved and mail flow is continuing as normal.

    If you continue to receive errors please open up a support ticket.

    This issue did not effect our backup mail servers.

  • Date - 09/01/2015 15:25 - 12/01/2015 00:00
  • Last Updated - 12/01/2015 15:28
Network Card Replacement - FS#7863 (Resolved)
  • Priority - High
  • Affecting Other - Network
  • We have detected packet loss from our upsteam provider. We have notified them of the issue and a fix is in place.
    Network packet loss was detected at 05/10/2014 02:47:37PM

    Quote from our provider:
    "A network component is currently malfunctioning, our team is currently in the process of resolving the issue."

    No ETA as of present, quick fix was to disable our edge firewall.

    05/10/2014 03:03:50PM
    • Packet loss
    • 05/10/2014 03:03:19PM
    • Los Angeles, CA
    • traceroute to 142.4.211.221 (142.4.211.221), 30 hops max, 60 byte packets
    • 1 204.152.200.41 (204.152.200.41) 38.268 ms 38.506 ms 38.606 ms
    • 2 colo-lax9.as29761.net (96.44.180.5) 1.229 ms 1.510 ms 1.613 ms
    • 3 * * *
    • 4 * * *
    • 5 bhs-g2-6k.qc.ca (198.27.73.207) 69.163 ms 69.413 ms 69.519 ms
    • 6 * * *
    • 7 198.27.73.244 (198.27.73.244) 96.413 ms 95.940 ms 96.386 ms
    • 8 vac3-0-a9.qc.ca.vaccum (198.27.73.245) 878.145 ms 875.964 ms 876.243 ms
    • 9 198.27.73.244 (198.27.73.244) 105.424 ms 105.612 ms 105.324 ms
    • 10 vac3-0-a9.qc.ca.vaccum (198.27.73.245) 887.881 ms * *
    • 11 198.27.73.244 (198.27.73.244) 115.666 ms * 114.575 ms
    • 12 * * *
    • 13 * * *
    • 14 * * *
    • 15 * * *
    • 16 * * *
    • 17 * * *
    • 18 * * *
    • 19 198.27.73.244 (198.27.73.244) 162.096 ms 162.341 ms *
    • 20 * * *
    • 21 * * *
    • 22 * * *
    • 23 * * *
    • 24 * * *
    • 25 * * *
    • 26 * * *
    • 27 * * *
    • 28 * * *
    • 29 * * *
    • 30 * * *

     
    Date and time of analysis: 05/10/2014 03:03:50PM
    • Packet loss
    • 05/10/2014 03:03:19PM
    • Las Vegas 4, NV

    • traceroute to 142.4.211.221 (142.4.211.221), 30 hops max, 60 byte packets
    • 1 72-46-153.static.versaweb.net (72.46.153.25) 0.194 ms 0.647 ms 1.317 ms
    • 2 192.228.109.49 (192.228.109.49) 0.971 ms 1.624 ms 1.893 ms
    • 3 any2ix.coresite.com (206.72.210.214) 7.733 ms * *
    • 4 * * *
    • 5 bhs-g2-6k.qc.ca (198.27.73.207) 79.350 ms 79.942 ms 80.188 ms
    • 6 * * *
    • 7 198.27.73.240 (198.27.73.240) 81.506 ms 81.840 ms 82.075 ms
    • 8 * vac3-0-a9.qc.ca.vaccum (198.27.73.241) 840.338 ms *
    • 9 198.27.73.240 (198.27.73.240) 86.970 ms * 86.049 ms
    • 10 * vac3-0-a9.qc.ca.vaccum (198.27.73.241) 841.078 ms *
    • 11 198.27.73.240 (198.27.73.240) 89.305 ms 90.774 ms 89.258 ms
    • 12 * * vac3-0-a9.qc.ca.vaccum (198.27.73.241) 859.539 ms
    • 13 * 198.27.73.240 (198.27.73.240) 96.724 ms 97.212 ms
    • 14 * * *
    • 15 * * 198.27.73.240 (198.27.73.240) 99.446 ms
    • 16 * * *
    • 17 198.27.73.240 (198.27.73.240) 102.100 ms 102.505 ms 102.832 ms
    • 18 * * *
    • 19 * * 198.27.73.240 (198.27.73.240) 105.513 ms
    • 20 * * *
    • 21 198.27.73.240 (198.27.73.240) 111.665 ms 113.031 ms 113.365 ms
    • 22 * * *
    • 23 * * *
    • 24 * * *
    • 25 * * *
    • 26 * * *
    • 27 * * *
    • 28 * * *
    • 29 * * *
    • 30 * * *
    Replacement installed.

  • Date - 05/10/2014 14:47 - 07/10/2014 14:11
  • Last Updated - 08/10/2014 19:06