Option `--rmi` doesn't remove images when used together with `-p`

[sylencecc]

Description
The command docker-compose -p project down --rmi local (or --rmi all ) won't remove unused images. This worked fine until compose 1.29.2 and fails with the current v2.6.1 as well as 2.1.0 (couldn't find a download to test with 2.0.0).

Steps to reproduce the issue:

1. Create minimal Dockerfile and docker-compose.yml:

$ cat Dockerfile
FROM alpine:3.16

$ cat docker-compose.yaml
services:

  api:
    build: .
    command: sleep infinity

3. Create the container with docker-compose -p abc up -d
4. After the container is running, remove it with docker-compose -p abc down --rmi local. Note that the image abc_api won't be removed.

 ⠿ Container abc-api-1  Removed
 ⠿ Network abc_default  Removed 

5. When running docker-compose -p abc down --rmi local (or --rmi all) again, the following warning appears. The image is still not removed.

Warning: No resource found to remove for project "abc".

Describe the results you received:
The docker images are kept despite telling docker-compose to remove them with --rmi local, manual removal with docker rmi or similar works. Removal works fine if no custom project name is specified with -p <name>.

Describe the results you expected:
(Local) images created during the build process should be removed when using --rmi local, even if a custom project name is specified with -p.

Output of docker compose version:

Docker Compose version v2.6.1

Reproducible with 2.1.0. Issue doesn't occur on 1.29.2 and earlier.

**Output of `docker info`:**
Client:
 Context:    default
 Debug Mode: false

Server:
 Containers: 15
  Running: 2
  Paused: 0
  Stopped: 13
 Images: 270
 Server Version: 20.10.12
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3df54a852345ae127d1fa3092b95168e4a88e2f8
 runc version: f46b6ba2c9314cfc8caae24a32ec5fe9ef1059fe
 init version: de40ad007797e0dcd8b7126f27bb87401d224240
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.15.41-gentoo.shade.v2
 Operating System: Gentoo Linux
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 7.775GiB
 Name: shade
 ID: MRH7:SMLW:H2RM:6K3D:GKL3:FNPB:O3NA:CQRZ:IFVS:W4JD:K6CU:GRXP
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  192.168.178.55:5000
  127.0.0.0/8
 Live Restore Enabled: false