Apple M1 or ARM64 support #476

master131 · 2021-10-27T09:12:23Z

Wanted to know if there was any intentions to make a arm64 compatible container available particularly for M1 Mac users who want to spin up an MQ instance?

At the company I work at, they've started to ship out M1 Macs to new starters and none of them have been able to successfully start/create a queue manager using the in-built amd64 emulation provided by Docker (using qemu). Previously we've been using Intel Macs which are natively running amd64 processor so there's no issues there.

Error 71 creating queue manager: AMQ8101S: IBM MQ error (893) has occurred.

See the below which is reporting the same issue:
docker/for-mac#5961

The text was updated successfully, but these errors were encountered:

arthurbarr · 2021-10-27T09:57:21Z

No immediate plans to do this. I think this is request would be better off submitted as a formal idea for MQ: https://ibm.biz/mqideas

There is some precedent for using MQ on ARM for the Raspberry Pi, but that won't directly translate into a container image.

Solo-Ranker · 2021-11-12T02:23:39Z

I've faced the same problem with M1 Mac. Our company is heavily relied on the IBM ecosystem. Therefore, we have to find other alternative MQ to run in the development environment.

ckruczek · 2021-11-18T13:34:22Z

I've also faced the same problem. The only way to get around this is currently to run a virtual machine with an intel docker image or changing our mac book which is not acceptable. If IBM could just provide a pre built tar for the mq for developers I could build my own image.

Neoklosch · 2021-11-18T14:49:49Z

Same here.
We also started to ship M1 Macs to our new employees.
Even in a long run, Apple will only provide M1s and at some point the Intel based Macs will be the minority.
So the need for an ARM based image will increase over time.

Mr-Light · 2021-12-11T11:28:45Z

Question, I have the same issue where IBM MQ does not work in docker, but I can't get past the java virtual machine error that this issue:

mq on M1

does not seem to be getting. Any ideas why?

Log where error occurs:

2021-12-11T11:04:38.901Z Creating queue manager QM1
2021-12-11T11:04:38.901Z Starting web server
2021-12-11T11:04:40.174Z Error 22 starting web server: 
Error: Port Library failed to initialize: -1
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

insiddius · 2022-01-09T02:39:37Z

Same here, any updates?

ghost · 2022-03-21T12:01:56Z

We have this issue too. Please concider a M1 version. (we moved also from intel)

This issue is really interesting for building up a local dev environment.

Kind Regards :-)

kassymbekoff · 2022-03-24T07:57:06Z

Still have this issue too

denismaggior8 · 2022-04-27T09:11:44Z

Same here... please add the porting to amr64 arch. on your roadmap

jannnci · 2022-05-11T13:48:48Z

same here, looking forward to pull docker with arm64 architecture

arthurbarr · 2022-05-17T10:45:39Z

Here's a link to the formal idea (RFE) for MQ software to support ARM processors: https://integration-development.ideas.ibm.com/ideas/MESNS-I-369

Adding votes on that idea is the most effective way to signal your support for this feature, which needs to be added to the base MQ software before we can then use it containers.

piers-walter-ibm · 2022-07-29T14:21:14Z

For those of you wanting to run MQ on M1, I've written a blogpost here with instructions on how to do it with Podman. It will be slower than native due to using x86_64 emulation but it's better than nothing

Schwaller · 2023-02-20T10:47:03Z

I need a solution that works with Testcontainers. The emulation for AMD on M2 works without issues for all images i've come across so I don't think its the emulation that breaks but the IBM MQ believes "there are multiple instances running" and shuts down. Can this mechanism be turned off somehow?

CPU architecture: amd64
Linux kernel version: 5.15.49-linuxkit
Base image: Red Hat Enterprise Linux 8.5 (Ootpa)
Running as user ID 1001 with primary group 0, and supplementary groups 0
Capabilities (bounding set): chown,dac_override,fowner,fsetid,kill,setgid,setuid,setpcap,net_bind_service,net_raw,sys_chroot,mknod,audit_write,setfcap
seccomp enforcing mode: filtering
Process security attributes: none
No volume detected. Persistent messages may be lost
You cannot run more than one instance of this program

Schwaller · 2023-02-28T08:53:50Z

Sooooooooooo ... is there any intention to support ARM any time soon? I've tried building the image for ARM myself but seems rather complicated. Isn't there somebody out there that did IBMs job for them already? Any known ARM image versions?

Schwaller · 2023-02-28T10:20:51Z

BTW the problem also appears when using the S390x architecture:

File docker-compose.yml:

services:
ibmMQ:
image: ibmcom/mq:9.2.4.0-r1
platform: linux/s390x
ports:
- "1414:1414"
- "9443:9443"
environment:
MQ_QMGR_NAME: "QM"
LICENSE: "accept"
MQ_ENABLE_METRICS : "false"
MQ_DEV: "true"
MQ_ADMIN_NAME: "admin"
MQ_ADMIN_PASSWORD: "incentage"
volumes:
- ./docker-ibm-mq-config.mqsc:/etc/mqm/20-config.mqsc
- ./ibmcom-data:/mnt/mqm

Result:

2023-02-28 11:17:03 2023-02-28T10:17:03.421Z CPU architecture: s390x
2023-02-28 11:17:03 2023-02-28T10:17:03.425Z Linux kernel version: 5.15.49-linuxkit
2023-02-28 11:17:03 2023-02-28T10:17:03.426Z Base image: Red Hat Enterprise Linux 8.5 (Ootpa)
2023-02-28 11:17:03 2023-02-28T10:17:03.427Z Running as user ID 1001 with primary group 0, and supplementary groups 0
2023-02-28 11:17:03 2023-02-28T10:17:03.428Z Capabilities (bounding set): chown,dac_override,fowner,fsetid,kill,setgid,setuid,setpcap,net_bind_service,net_raw,sys_chroot,mknod,audit_write,setfcap
2023-02-28 11:17:03 2023-02-28T10:17:03.429Z seccomp enforcing mode: filtering
2023-02-28 11:17:03 2023-02-28T10:17:03.429Z Process security attributes: none
2023-02-28 11:17:03 2023-02-28T10:17:03.429Z Detected 'fuse.grpcfuse' volume mounted to /mnt/mqm
2023-02-28 11:17:03 2023-02-28T10:17:03.732Z You cannot run more than one instance of this program

master131 · 2023-04-14T05:14:28Z

Multiple instances issue is due to the following function in process.go compiled as part of the container image:

mq-container/cmd/runmqserver/process.go

Line 48 in 53790e1

numOfProg := strings.Count(out, programName)

It does a simple count of the number of times that runmqdevserver appears in the ps -ef --format cmd command output. However, when the image is run under emulation on M1/M2 Mac machines, the ps listing looks like this:

/usr/bin/qemu-x86_x64 /usr/local/bin/runmqdevserver runmqdevserver

Subsequently it thinks its detected more than 1 instance of runmqdevserver. However, even if you fix this issue (ie. by reading the output line by line) you will still run into this error:

Error 71 creating queue manager: AMQ8101S: IBM MQ error (893) has occurred.

It seems there is some issue creating the queue manager and can be minimally reproduced as per below:

docker run -it --platform linux/amd64 --entrypoint /bin/sh ibmcom/mq:latest
crtmqm -c ABC QM1

The FDC files can be obtained accordingly using docker cp for inspection.

+-----------------------------------------------------------------------------+
|                                                                             |
| IBM MQ First Failure Symptom Report                                         | 
| =========================================                                   | 
|                                                                             |
| Date/Time         :- Fri April 14 2023 05:25:45 UTC                         | 
| UTC Time          :- 1681449945.779745                                      | 
| UTC Time Offset   :- 0 (UTC)                                                | 
| UTC Date/Time     :- 2023-04-14 05:25:45.779745 (1681449945779745)          | 
| Host Name         :- d7cee84f65bc                                           | 
| Operating System  :- Linux 5.15.49-linuxkit                                 | 
| OS Details        :- Red Hat Enterprise Linux 8.5 (Ootpa)                   | 
| PIDS              :- 5724H7251                                              | 
| LVLS              :- 9.2.4.0                                                | 
| Product Long Name :- IBM MQ for Linux (x86-64 platform)                     | 
| Vendor            :- IBM                                                    | 
| O/S Registered    :- 0 (libmqmcs_r.so) Unzipped                             | 
| Data Path         :- /mnt/mqm/data                                          | 
| Installation Path :- /opt/mqm                                               | 
| Installation Name :- Installation1    (126)                                 | 
| License Type      :- Developer                                              | 
| Probe Id          :- HL000001                                               | 
| Application Name  :- MQM                                                    | 
| Component         :- hlgStartLog                                            | 
| Source filename   :- /build/slot1/p920_P/src/lib/logger/amqhlgc0.c          | 
| Line Number       :- 656                                                    | 
| Build Date        :- Nov  5 2021                                            | 
| Build Level       :- p924-L211105.DE                                        | 
| Build Type        :- IKAP - (Production)                                    | 
| Effective UserID  :- 1001 (assumed to be mqm)                               | 
| Real UserID       :- 1001 (assumed to be mqm)                               | 
| Program Name      :- crtmqm                                                 | 
| Arguments         :- -c ABC QM1                                             | 
| Addressing mode   :- 64-bit                                                 | 
| LANG              :- en_US.UTF-8                                            | 
| Process           :- 67                                                     | 
| Process(Thread)   :- 67                                                     | 
| Thread            :- 1    Unknown                                           | 
| QueueManager      :- QM1                                                    | 
| SubpoolName       :- QM1                                                    | 
| Perf Counter      :- 517275518734                                           | 
| UserApp           :- FALSE                                                  | 
| ConnId(2) QM      :- 2                                                      | 
| Last HQC          :- 2.0.0-705408                                           | 
| Last HSHMEMB      :- 2.4.0-4478656                                          | 
| Last ObjectName   :-                                                        | 
| Major Errorcode   :- xecF_E_UNEXPECTED_RC                                   | 
| Minor Errorcode   :- hrcE_MQLO_SEMA                                         | 
| Probe Type        :- MSGAMQ6118                                             | 
| Probe Severity    :- 1                                                      | 
| Probe Description :- AMQ6118S: An internal IBM MQ error has occurred        |
|   (40006118)                                                                | 
| FDCSequenceNumber :- 0                                                      | 
| Arith1            :- 545286149 (0x20806805)                                 | 
|                                                                             |
+-----------------------------------------------------------------------------+

With emulation runmqdevserver appears multiple times in a single line of `ps` output, so we need to count the number of lines with runmqdevserver instead. Issue: ibm-messaging#476

adamdecaf · 2023-05-04T15:34:10Z

@master131 how did you get to the AMQ8101S: IBM MQ error (893) has occurred. error? Were you using an M1 mac and emulating the processes? I'm not seeing MQ Advanced for Developers released for ARM/M1 chips.

https://github.com/ibm-messaging/mq-container/blob/master/docs/building.md#building-a-developer-image

With emulation runmqdevserver appears multiple times in a single line of `ps` output, so we need to count the number of lines with runmqdevserver instead. Issue: ibm-messaging#476

adamdecaf · 2023-05-04T21:41:41Z

I've captured some error logs from running my patched container.

AMQ119.0.FDC.txt
MQERR01.LOG
AMQERR01.json.txt

Inside I am seeing some strange logs, but am unfamiliar if they're causing issues.

Q: Should this first file descriptor point to /dev/null? That seems wrong.

File Descriptors
lr-x------ 1 1001 0 64 2023-05-04 20:38:25.143804 /proc/119/fd/0 -> crw-rw-rw- 1 1001 0 64 2023-05-04 20:37:42.463449 /dev/null
l-wx------ 1 1001 0 64 2023-05-04 20:38:25.143804 /proc/119/fd/1 -> pipe:[40812]
l-wx------ 1 1001 0 64 2023-05-04 20:38:25.143804 /proc/119/fd/2 -> pipe:[40812]
lrwx------ 1 1001 0 64 2023-05-04 20:38:25.143804 /proc/119/fd/3 -> -rw-rw-r-- 1 1001 0 64 2023-05-04 20:38:09.404449 /mnt/mqm/data/sockets/@SYSTEM/qmgrlocl/dd34396f3287/Installation1/Trace.shm
lrwx------ 1 1001 0 64 2023-05-04 20:38:25.143804 /proc/119/fd/4 -> -rw-rw-r-- 1 1001 0 64 2023-05-04 20:38:19.847804 /mnt/mqm/data/qmgrs/QM1/master
lrwx------ 1 1001 0 64 2023-05-04 20:38:25.143804 /proc/119/fd/5 -> -rw-rw---- 1 1001 0 64 2023-05-04 20:38:19.822804 /mnt/mqm/data/sockets/QM1/qmgrlocl/dd34396f3287/Enqueue.lck
lr-x------ 1 1001 0 64 2023-05-04 20:38:25.143804 /proc/119/fd/6 -> -rw-rw-r-- 1 1001 0 64 2023-05-04 20:38:19.848804 /mnt/mqm/data/qmgrs/QM1/active
l-wx------ 1 1001 0 64 2023-05-04 20:38:25.143804 /proc/119/fd/7 -> -rw-rw---- 1 1001 0 64 2023-05-04 20:38:25.144804 /mnt/mqm/data/errors/AMQ119.0.FDC
lr-x------ 1 1001 0 64 2023-05-04 20:38:25.143804 /proc/119/fd/8 -> dr-x------ 1 1001 0 64 2023-05-04 20:38:25.143804 /proc/119/fd

| Major Errorcode   :- xecF_E_UNEXPECTED_RC                                   | 
| Minor Errorcode   :- hrcE_UNEXPECTED_ERROR                                  | 
| Probe Type        :- MSGAMQ6118                                             | 
| Probe Severity    :- 1                                                      | 
| Probe Description :- AMQ6118S: An internal IBM MQ error has occurred        |
|   (40006118)                                                                |

In #327 (comment) I see that /mnt/mqm (and other directories) have mqm/mqm permissions, but the M1 mac has 1001/root for the user/group. Would those permissions be cause crtmqm to exit with a 71 code?

master131 · 2023-05-05T09:15:41Z

@master131 how did you get to the AMQ8101S: IBM MQ error (893) has occurred. error? Were you using an M1 mac and emulating the processes? I'm not seeing MQ Advanced for Developers released for ARM/M1 chips.

https://github.com/ibm-messaging/mq-container/blob/master/docs/building.md#building-a-developer-image

Was just running the x86_64 version via docker under emulation on M1 Mac use command earlier (with a fix to the runmqserver).

master131 · 2023-05-05T09:18:59Z

In #327 (comment) I see that /mnt/mqm (and other directories) have mqm/mqm permissions, but the M1 mac has 1001/root for the user/group. Would those permissions be cause crtmqm to exit with a 71 code?

The changelog states:

Can now run as a random user, instead of the "mqm" user, which has now been removed. This adds compatability for the Red Hat OpenShift restricted SCC. The default image UID is 1001.

Schwaller · 2023-05-05T10:56:02Z

So its working for you with some latest version? Could you please be a little more specific how we can run IBM MQ on M1?
I guess you followed this https://github.com/ibm-messaging/mq-container/blob/master/docs/building.md#building-a-developer-image but wich branch/version?

Thanks to everyone working on this!

master131 · 2023-05-05T11:04:46Z

So its working for you with some latest version?

No, it's not - not unless you resort to workarounds that have already been documented online such as using podman or UTM:

https://medium.com/@thejackyhuang/mq-dev-on-m1-with-utm-8b3c90403ef8
https://community.ibm.com/community/user/integration/blogs/piers-walter/2022/07/28/running-x86-64-containers-such-as-mq-on-podman-on?CommunityKey=183ec850-4947-49c8-9a2e-8e7c7fc46c64

Schwaller · 2023-05-05T11:06:10Z

Ah ... sorry just woke up and got exited ^^ Still thanks for working on it :)
I don't understand how the x86 emulation can lead to this kind of problems but using a virtual machine somehow works as expected. Something very strange going on. Also nice catch with that ps output issue.

adamdecaf · 2023-05-05T14:07:07Z

I'm building an x86 image on my intel mac and running it emulated on my M1 mac. I've got #524 to get past the first error, but am seeing AMQ8101S: IBM MQ error (893) has occurred. still.

Is anyone more familiar with MQ errors to know how to work around that? I have full logs posted in #476 (comment)

With emulation runmqdevserver appears multiple times in a single line of `ps` output, so we need to count the number of lines with runmqdevserver instead. Issue: ibm-messaging#476

During emulation the running processes can contain binary names which trip up detecting if a process is already running. Using the comm format will print the running binaries without arguments. Previous: ibm-messaging#524 Issue: ibm-messaging#476

chughts · 2023-06-29T16:59:28Z

The building.md documentation in this repo, has been updated to reflect that it's now possible to build your own container for both amd64 and arm64 (Apple Silicon) architectures.

https://github.com/ibm-messaging/mq-container/blob/master/docs/building.md#building-a-developer-image

pascaldekloe · 2023-07-01T19:15:15Z

Works perfectly out-of-the-box. 👏

% docker ps
CONTAINER ID   IMAGE                                     COMMAND                  CREATED             STATUS                        PORTS                                                                         NAMES
29c832bb6c81   ibm-mqadvanced-server-dev:9.3.3.0-arm64   "runmqdevserver"         3 minutes ago       Up About a minute             1414/tcp, 9157/tcp, 9443/tcp

master131 · 2023-07-02T00:47:57Z

Awesome news guys, I am closing this issue.

koyomi-san · 2023-08-16T07:37:29Z

hello @master131 @pascaldekloe, how did you managed to run it on an m1 or m2 chip? could you give me a hand? thank you

pascaldekloe · 2023-08-16T11:15:45Z

Did you read building.md as suggested by chughts @koyomi-san?

koyomi-san · 2023-08-16T14:09:18Z

@pascaldekloe yes, also found this instructions:
https://community.ibm.com/community/user/integration/blogs/richard-coppen/2023/06/30/ibm-mq-9330-container-image-now-available-for-appl
thanks for your help, it worked for me.

This was referenced Mar 29, 2022

You cannot run more than one instance of this program #492

Closed

Error java Mac M1 #480

Closed

pascaldekloe mentioned this issue Mar 19, 2023

IBM MQ in integration/docker-compose.yml pascaldekloe/mqtt#4

Open

adamdecaf mentioned this issue May 4, 2023

fix: count instances of runmqdevserver correctly on M1 macs #524

Closed

adamdecaf mentioned this issue Jun 9, 2023

fix: use better "ps" format #527

Open

master131 closed this as completed Jul 2, 2023

arthurbarr mentioned this issue May 28, 2024

Running on ARM exits without any error message #562

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Apple M1 or ARM64 support #476

Apple M1 or ARM64 support #476

master131 commented Oct 27, 2021

arthurbarr commented Oct 27, 2021

Solo-Ranker commented Nov 12, 2021

ckruczek commented Nov 18, 2021

Neoklosch commented Nov 18, 2021

Mr-Light commented Dec 11, 2021 •

edited

Loading

insiddius commented Jan 9, 2022

ghost commented Mar 21, 2022

kassymbekoff commented Mar 24, 2022

denismaggior8 commented Apr 27, 2022

jannnci commented May 11, 2022

arthurbarr commented May 17, 2022

piers-walter-ibm commented Jul 29, 2022

Schwaller commented Feb 20, 2023 •

edited

Loading

Schwaller commented Feb 28, 2023

Schwaller commented Feb 28, 2023 •

edited

Loading

master131 commented Apr 14, 2023 •

edited

Loading

adamdecaf commented May 4, 2023

adamdecaf commented May 4, 2023 •

edited

Loading

master131 commented May 5, 2023 •

edited

Loading

master131 commented May 5, 2023

Schwaller commented May 5, 2023 •

edited

Loading

master131 commented May 5, 2023 •

edited

Loading

Schwaller commented May 5, 2023 •

edited

Loading

adamdecaf commented May 5, 2023

chughts commented Jun 29, 2023

pascaldekloe commented Jul 1, 2023

master131 commented Jul 2, 2023

koyomi-san commented Aug 16, 2023

pascaldekloe commented Aug 16, 2023

koyomi-san commented Aug 16, 2023

Apple M1 or ARM64 support #476

Apple M1 or ARM64 support #476

Comments

master131 commented Oct 27, 2021

arthurbarr commented Oct 27, 2021

Solo-Ranker commented Nov 12, 2021

ckruczek commented Nov 18, 2021

Neoklosch commented Nov 18, 2021

Mr-Light commented Dec 11, 2021 • edited Loading

insiddius commented Jan 9, 2022

ghost commented Mar 21, 2022

kassymbekoff commented Mar 24, 2022

denismaggior8 commented Apr 27, 2022

jannnci commented May 11, 2022

arthurbarr commented May 17, 2022

piers-walter-ibm commented Jul 29, 2022

Schwaller commented Feb 20, 2023 • edited Loading

Schwaller commented Feb 28, 2023

Schwaller commented Feb 28, 2023 • edited Loading

master131 commented Apr 14, 2023 • edited Loading

adamdecaf commented May 4, 2023

adamdecaf commented May 4, 2023 • edited Loading

master131 commented May 5, 2023 • edited Loading

master131 commented May 5, 2023

Schwaller commented May 5, 2023 • edited Loading

master131 commented May 5, 2023 • edited Loading

Schwaller commented May 5, 2023 • edited Loading

adamdecaf commented May 5, 2023

chughts commented Jun 29, 2023

pascaldekloe commented Jul 1, 2023

master131 commented Jul 2, 2023

koyomi-san commented Aug 16, 2023

pascaldekloe commented Aug 16, 2023

koyomi-san commented Aug 16, 2023

Mr-Light commented Dec 11, 2021 •

edited

Loading

Schwaller commented Feb 20, 2023 •

edited

Loading

Schwaller commented Feb 28, 2023 •

edited

Loading

master131 commented Apr 14, 2023 •

edited

Loading

adamdecaf commented May 4, 2023 •

edited

Loading

master131 commented May 5, 2023 •

edited

Loading

Schwaller commented May 5, 2023 •

edited

Loading

master131 commented May 5, 2023 •

edited

Loading

Schwaller commented May 5, 2023 •

edited

Loading